Assessing evidence: strengths and weaknesses of grading tools

Resource Type: 

Grading instruments can help policy decision makers evaluate research on a given topic. There are many grading instruments available to “…provide a metric to ‘quantify’ the quality of evidence from either an individual study or from a body of evidence”. Their strengths are that they are systematic, time effective, accessible, objective and allow clear communication. However, grading instruments can fail to adequately represent the quality of research, with problems including:

  • “Lack of validation and poor reliability
  • Highly subjective – low interrater reliability
  • Create false sense of trust or over-reliance on grading outcomes
  • Overly complicated, unclear instructions and little or no training – mistakes made
  • Biased in favour of certain types of studies (eg., randomised control trials) and against others (eg., observational).
  • Grading of evidence may not be applicable to local settings.”

An example from the group’s own research on smoking as a risk factor for Alzheimer’s Disease demonstrates the problems. Observational research findings were rated poorly even though randomised controlled trials would have been unethical.

Greater awareness of short-comings is needed, along with critical evaluation of scores. There should also be better matching of grading instruments with particular types of evidence, along with the development of improved methods.

Reference: Irving, M., Cherbuin, N., Eramudugolla, R., Butterworth, P., O’Donoghue Jenkins, L. and Anstey, K. (2013). Tools for Grading Evidence: Strengths, Weaknesses, and the Impact on Effective Knowledge Translation, digital poster #736, from the First Global Conference on Research Integration and Implementation held in Canberra, Australia, online and at three co-conferences (Lueneburg in Germany, The Hague in the Netherlands and Montevideo in Uruguay), 8-11 September 2013.

You can access this digital poster as a powerpoint presentation or pdf at: http://i2sconference.digitalposter.com.au/posters-list/736.

Posted: December 2013

Last modified: January 2015

About this site Updated:  20 April 2017/Responsible Officer:    Manager I2S/Page Contact:    Webmaster