JBI releases revised Critical Appraisal Tools
Focusing on risk of bias assessment
For almost 25 years, JBI’s critical appraisal tools have assisted systematic reviewers assess the trustworthiness, relevance and results of published papers and have undergone many different iterations as the field of evidence synthesis has continued to evolve.
This latest iteration of JBI’s critical appraisal tools ensures that the tools remain current and are aligned with current methodological developments and the evolution of terminology in the field. Dr Tim Barker, lead author of Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process outlines what has changed and how JBI’s critical appraisal tools are unique in an interview.
What are critical appraisal tools?
Historically, critical appraisal tools were educational instruments used by clinicians and researchers to evaluate the quality of research papers. They have been adopted for use in systematic reviews and facilitate the assessment of a research study’s limitations. One domain of critical appraisal involves the evaluation of study bias.
What is risk of bias?
The assessment of a study’s risk of bias helps researchers determine the internal validity of that study (i.e., was that study free from systematic error and therefore are the results presented likely to be true). A research study can be at risk of bias in its conduct, analysis and interpretation of its data. The assessment of a study’s risk of bias is a fundamental step in the systematic review of quantitative evidence.
Why were JBI’s critical appraisal tools revised?
As evidence-based healthcare is constantly evolving it has been recently made important that when we conduct systematic reviews (of quantitative evidence) that we only consider the risk of bias of our included studies and no other constructs of quality. The previous iterations of the JBI critical appraisal tools for quantitative study designs did not always ask questions that were strictly related to a study’s risk of bias. While the revised tools still ask questions that do not relate to risk of bias, they have been clearly separated from those that do, to support users of these revised instruments in determining both a study’s risk of bias, as well as other validity constructs.
What has changed?
In addition to the above mentioned ‘separation’ of questions based on the construct of quality they are related to, we have also updated the tool to:
[1] facilitate decision making at the outcome and result level and
[2] organise questions related to risk of bias to the domain of bias they belong.
[1] When we assess the risk of bias of an included study, it is important that we do so at the appropriate level. For example, a question related to the randomisation of participants included in an RCT is answered at the study level, meaning that this result holds true for every outcome and result included in that study. However, a question about the reliability of an outcome measure can only be answered at the outcome level, as the reliability of an outcome measurement changes for each outcome included in the RCT. The previous iterations of the JBI critical appraisal instruments only allowed assessments at the study level, which was not aligned to current best-practice for how risk of bias assessment should be performed.
[2] Each question in the revised tool related to risk of bias is now aligned to a specific domain of bias. Multiple different questions may be asked in a critical appraisal tool that each have an important bearing on determining that study’s risk of bias. A question about the randomisation process may be presented as well as a question on the process of how the allocation of that randomisation sequence was concealed from key individuals in the study. Both questions are related to fundamentally different aspects of the design of the study, however both are related to the domain of bias related to how participants have been selected in the study. The previous iterations of the tool did not present the questions aligned in such a manner. We believe that by doing this, users of these instruments will be able to better discuss their risk of bias assessments in terms of these overarching domains.
How are JBI’s tools unique?
The JBI tools are unique in they are not prescriptive in the methodological approach needed to be followed when they are used. The design of some critical appraisal tools force the users to adopt a particular approach in how the process of critical appraisal is conducted. For example, the Downs and Black checklist is structured for its users to adopt a “scale-based” approach, where its users assign a score of 0-2 to each question presented, which is then numerated for an overall quality score. Cochrane’s risk of bias 2.0 tool has been designed for its users to follow a “domain-based” approach, where judgements are made across a specific domain of bias and an overall risk of bias is made using qualitative judgments of value. There is no evidence to suggest that one approach is better than the other.
The JBI tools have been very specifically designed to facilitate any approach that the review team chooses, which makes them more adaptable to the competencies and skill of the review team.
Access the revised JBI critical appraisal tools at jbi.global/critical-appraisal-tools
Publications on the revised tools:
Assessing the risk of bias of quantitative analytical studies: introducing the vision for critical appraisal within JBI systematic reviews