Critically appraising diverse textual sources
Tools for narrative, expert opinion and policy evidence
A recent publication in JBI Evidence Synthesis outlines updated methodological guidance for critically appraising textual evidence in systematic reviews, moving from a single critical appraisal tool to three distinct tools for narrative, expert opinion, and policy texts.
Textual evidence systematic reviews series
The publication, titled Textual evidence systematic reviews series paper 3: critical appraisal of evidence from narrative, opinion, and policy, is the final paper in a series of three. The series was launched with the publication of the first paper, an introduction to the revised JBI methodology and overview of recent changes, in the special methodology issue of JBI Evidence Synthesis to mark JBI Methodology Month in March 2025.
The series underscores the growing recognition and methodological advancements in the synthesis of non-research evidence, particularly textual evidence, to inform healthcare policy and practice.
Appraising different types of text as evidence
Evidence-based practice values an inclusive view of what counts as evidence, extending beyond traditional research studies. Systematic reviews, which gather and evaluate evidence on a topic, increasingly include various forms of text, such as narrative accounts, expert opinions, and policy documents. Critically appraising these non-research sources is essential for ensuring the reliability of information presented.
The critical appraisal of non-research sources requires a nuanced understanding of the contextual relevance and the expertise of the contributors, highlighting the necessity for a more flexible and inclusive approach to critical appraisal.
Unlike empirical research, these textual sources often lack standardised methods, which can make assessing their quality challenging. The subjective nature of opinions or the political influences in policies can introduce biases and complexity. To effectively evaluate such diverse evidence, a flexible and nuanced approach is necessary.
Updated JBI Critical Appraisal Tools
Recognising the growing use and the unique nature of different textual sources, JBI has updated its methodology for critically appraising textual evidence. Previously, one general tool was used for narrative, opinion, and policy. Now, following a process involving international experts, three separate critical appraisal tools have been developed: one for narrative, one for expert opinion, and one for policy text.
This updated approach is informed by principles of discourse analysis and models such as Toulmin's Model of Argumentation, which help break down and assess the structure and legitimacy of arguments presented in the text.
Tailored assessment
The development of separate tools highlights the unique characteristics of each data source and provides specific guidance for reviewers to interrogate their legitimacy and authenticity effectively.
For narrative evidence, the tool focuses on the credibility of the source, the context of the account, the logical flow of events, and the distinction between narrative and "storytelling".
When appraising expert opinion, the tool prompts reviewers to examine the source's standing in the field, the logical defence of the argument, the reference to existing literature, and awareness of alternate views.
For policy evidence, the appraisal considers the identification and standing of developers, the involvement of relevant stakeholders, acknowledgment of biases, and the process used to gather evidence and develop recommendations.
The 3 critical appraisal tools for textual evidence are available with 12 other tools for different types of studies.
This refined methodology provides a more robust way to incorporate and evaluate textual evidence, acknowledging its significant value alongside other evidence types in informing healthcare and policy decisions.
Resources for textual evidence synthesis methodology
The full series of papers offers supplementary support for conducting high-quality systematic reviews of textual evidence. Comprehensive, step-by-step- guidance for conducting systematic reviews of textual evidence is available in the JBI Manual for Evidence Synthesis.
Paper 3 is available in the May 2025 issue of JBI Evidence Synthesis:
McArthur, Alexa; Cooper, Adam; Edwards, Deborah; Klugarova, Jitka; Yan, Hu; Barber, Brittany V.; Gregg, Emily E.; Weeks, Lori E.; Jordan, Zoe
JBI Evidence Synthesis. 23(5):833-839, May 2025.