6.1 Critical evaluation of sources
Critical appraisal has been described as "the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context" (Burls, 2009).
Has the research been conducted in a way that minimises bias? (Is it trustworthy?)
What is the impact and importance of the findings? (What is the research's value?)
Are the results applicable to your patient/population/problem? (Is it relevant to your research question?)
Use the quick links below to learn how to recognise what a suitable source is for academic study.
It is very easy to find information thanks to the convenience of the Internet. However, it can be difficult to know which source or which piece of information is reliable.
A reliable source:
is published by an expert in their field
presents information that has been researched and shows evidence of that research
has research that has been checked by another expert, or has gone through an editorial process
is objective. Its aim is to inform without any bias
is up-to-date
Rauru Whakarere Evaluation Framework
The Rauru Whakarere Evaluation Framework for assessing resources is a kaupapa Māori strategy. It involves five interconnected concepts:
1. Whakapapa (the background)
2. Orokohanga (the origins)
3. Mana (the authority)
4. Māramatanga (the content)
5. Aronga (the lens)
Learn more about each concept:
References
Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Research Synthesis Methods. 2019;10(3):330-342.
Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, Hartling L. Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Syst Rev. 2019 Nov 15;8(1):278. doi: 10.1186/s13643-019-1222-2. PMID: 31727150; PMCID: PMC6857345.
Screening is the process of identifying which studies from your literature search(es) will be included in your review. Each individual article must be assessed to see if it meets the inclusion criteria you've set out in your protocol.
Screening is a time-consuming two-part process to determine which citations meet the team's eligibility criteria and should be included in the review. To reduce bias, a minimum of two reviewers typically screen the search results and resolve areas of disagreement by consensus or a third tie-breaker team member who is a content expert. A number of online tools, such as Covidence or Rev Man, exist to facilitate the process.
A standard workflow for this stage might be:
The PRISMA flow diagram visually summarises the screening process. It makes the selection process transparent and clear to the reader by reporting on decisions made at various stages of the systematic review. When you're excluding articles at the full-text stage, you need to specify the reasons for exclusion. It's important to document this thoroughly.
Pre-screening: Record the number of results from each database or source recorded before screening commences
Deduplication: Remove duplicated articles from the combined result set.
Title/abstract screening: Each reviewer will need to read titles and abstracts to see if they match the inclusion/exclusion criteria or have some value to the systematic review. This is done separately by each reviewer to ensure minimal bias. Results are then compared.
Full-text screening: Included articles are located in full-text prior to multiple reviewers individually read through the full-text of included articles to fine-tune the final collection of articles that will contribute to the review.
Module 4: Selecting Studies and Collecting Data
*Please note you will need to register for a Cochrane account while initially on the Mayo network. You'll receive an email message containing a link to create a password and activate your account.*
Removal of duplications
After you have run your searches, you will need to remove duplicates from your complete list of results. You can do this in EndNote - see Find Duplicates in the EndNote library guide - or in Zotero - see Duplicate Detection from the Zotero website. Then you need to screen the results to find the most rigorous and appropriate to help answer your research question.
The reviewing team
To reduce the risk of selection errors or bias, you should have at least two reviewers (including yourself) to screen your search results.
Two stages of screening
1. Title/Abstract screening: You need to scan the titles and abstracts to remove the obviously irrelevant studies. At this stage you might not need to provide a justification for your exclusions.
2. Full-text screening: You need to locate the full text of the included studies and carefully examine each one for compliance with your eligibility criteria. The reviewers should document the reason for excluding any articles.
Your review paper should provide details of how many reviewers screened the articles and how any disagreements were resolved. This is usually in the Methods section.
For more information go to The Cochrane Handbook - Part 2, Chapter 4.6 (Selecting studies)
Tools you can use
Most of the tools listed in this guide in Tools and Software can be used for screening:
Referencing PRISMA
From the PRISMA website: "The PRISMA Statement and the PRISMA Explanation and Elaboration document are distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited." This means that you must reference PRISMA when you use it.
Undergraduate students:
You can locate all the information you need for the reference list on the flow diagram document. Format the information according to APA 7. It will usually be formatted as a journal article with a DOI.
For the in-text citation, use the below as a model and put it beneath your adapted PRISMA diagram:
Note. Adapted from Title, by Author, Year.
Postgraduate students: Click here for more information.
Covidence is an online software product designed to improve the efficiency and experience of creating and maintaining systematic reviews. Covidence will help you with the selection (screening) and extraction phases of your project.
The following resources will help you navigate Covidence: