There are generally four stages of a knowledge synthesis, each of which includes several tasks. Most of these tasks apply to all types of syntheses, but not all. For example, scoping reviews do not necessarily require quality assessment of studies.
Included on this page are:
Most types of knowledge syntheses cannot be conducted by one person. Ideally, you need a team of at least three people which includes:
Read these helpful Tips for a Successful Review Team from Northeastern University Library.
Developing clear and focused questions is crucial in a knowledge synthesis for several reasons:
Consider using one of the frameworks listed in the table below to help guide question development. (Adapted from: Foster, M. & Jewell, S. (Eds). (2017). Assembling the pieces of a systematic review: Guide for librarians. Medical Library Association, Lanham: Rowman & Littlefield. p. 38, Table 3.3.).
| Framework | Stands for |
Original Source Examples/templates (where available) |
Type of Question/ |
|---|---|---|---|
| BeHEMNoTh | Be: behavior of interest H: health contest (service/policy/intervention) E: exclusions MoTh: models or theories |
Booth, A., & Carroll, C. (2015). Systematic searching for theory to inform systematic reviews: Is it feasible? Is it desirable? Health Information and Libraries Journal, 32(3), 220–235. https://doi.org/10.1111/hir.12108 Examples/templates:
|
Questions about theories |
| CHIP | Context How Issues Population |
Shaw, R. (2010). Conducting literature reviews. In M. A. Forester (Ed.), Doing Qualitative Research in Psychology: A Practical Guide (pp. 39-52). London, Sage | Psychology, qualitative |
| CIMO | Context Intervention Mechanisms Outcomes |
Denyer, D., & Tranfield, D. (2009). Producing a systematic review. In D. A. Buchanan & A. Bryman (Eds.), The Sage handbook of organizational research methods (pp. 671-689). Thousand Oaks, CA: Sage Publications Ltd. | Management, business, administration |
| CLIP |
Client group |
Wildridge, V., & Bell, L. (2002). How CLIP became ECLIPSE: A mnemonic to assist in searching for health policy/management information. Health Information & Libraries Journal, 19(2), 113–115. https://doi.org/10.1046/j.1471-1842.2002.00378.x | Management, business, administration |
| CoCoPop |
Condition |
Munn Z, Moola S, Lisy K, Riitano D, Tufanaru C. Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data. Int J Evid Based Healthc. 2015 Sep;13(3):147-53. doi: 10.1097/XEB.0000000000000054. PMID: 26317388. Examples/templates:
|
Prevelance, incidence |
| COPES | Client-Oriented Practical Evidence Search |
Gibbs, L. (2003). Evidence-based practice for the helping professions: A practical guide with integrated multimedia. Pacific Grove, CA: Brooks/Cole-Thomson Learning. | Social work, health care, nursing |
| ECLIPSe | Expectation Client Location Impact Professionals Service |
Wildridge, V., & Bell, L. (2002). How CLIP became ECLIPSE: A mnemonic to assist in searching for health policy/management information. Health Information & Libraries Journal, 19(2), 113–115. https://doi.org/10.1046/j.1471-1842.2002.00378.x Examples/templates:
|
Management, services, policy, social care |
| PEO | Population Exposure Outcome |
Khan, K., Kunz, R., Kleijnen, J., & Antes, G. (2011). Systematic reviews to support evidence-based medicine : how to review and apply findings of healthcare research (Second edition.). CRC Press. Examples/templates:
|
Qualitative |
| PECODR | Patient/population/problem Exposure Comparison Outcome Duration Results |
Dawes, M., Pluye, P., Shea, L., Grad, R., Greenberg, A., & Nie, J.-Y. (2007). The identification of clinically important elements within medical journal abstracts: Patient_Population_Problem, Exposure_Intervention, Comparison, Outcome, Duration and Results (PECODR). Journal of Innovation in Health Informatics, 15(1), 9–16. |
Medicine |
| PerSPECTiF | Perspective Setting Phenomenon of interest/Problem Environment Comparison (optional) Time/Timing Findings |
Booth, A., Noyes, J., Flemming, K., Moore, G., Tunçalp, Ö., & Shakibazadeh, E. (2019). Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Global Health, 4(Suppl 1). Examples/templates:
|
Qualitative research |
| PESICO | Person Environments Stakeholders Intervention Comparison Outcome |
Schlosser, R. W., & O'Neil-Pirozzi, T. (2006). Problem formulation in evidence-based practice and systematic reviews. Contemporary Issues in Communication Sciences and Disorders, 33, 5-10. | Augmentative and alternative communication |
| PICO | Patient Intervention Comparison Outcome |
Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP journal club, 123(3), A12-A12. Examples/Templates:
|
Clinical medicine |
| PICO+ |
Patient +context, patient values, and preferences |
Bennett, S., & Bennett, J. W. (2000). The process of evidence‐based practice in occupational therapy: Informing clinical decisions. Australian Occupational Therapy Journal, 47(4), 171-180. | Occupational therapy |
| PICOC |
Patient Study Type |
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell Publishers. | Social Sciences |
| PICOS |
Patient Study Type |
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & Prisma Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine, 6(7), e1000097. | Medicine |
| PICOT |
Patient Time |
Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP journal club, 123(3), A12-A12. Examples/templates:
|
Education, health care |
| PICO specific to diagnostic tests | Patient/participants/population Index tests Comparator/reference tests Outcome |
Kim, K. W., Lee, J., Choi, S. H., Huh, J., & Park, S. H. (2015). Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: A practical review for clinical researchers - Part I. General guidance and tips. Korean Journal of Radiology, 16(6), 1175-1187. | Diagnostic questions |
| ProPheT | Problem Phenomenon of interest Time |
Booth, A., Noyes, J., Flemming, K., Gerhardus, A., Wahlster, P., van der Wilt, G. J., ... & Rehfuess, E. (2016). Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions. [Technical Report]. https://doi.org/10.13140/RG.2.1.2318.0562 ----- Booth, A., Sutton, A., & Papaioannou, D. (2016). Systematic approaches to a successful literature review (2. ed.). London: Sage. |
Social sciences, qualitative, library science |
| SPICE | Setting Perspective Interest Comparison Evaluation |
Booth, A. (2006). Clear and present questions: formulating questions for evidence based practice. Library Hi Tech, 24(3), 355-368. Examples/templates:
|
Library and information sciences |
| SPIDER | Sample Phenomenon of interest Design Evaluation Research type |
Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative health research, 22(10), 1435-1443. Examples/templates:
|
Health, qualitative research |
Once you have your question, you’ll need to look for existing reviews and ongoing protocols on your topic to make sure you’re not duplicating work. This may also help you make any necessary changes to your question to address other gaps in the research.
While searching for reviews you'll also be able to assess the volume of potentially relevant studies as well as identify key studies. Key studies can be used as 'seed articles' providing a starting point for developing comprehensive search strategies. They can also serve as a benchmark for assessing the success of your search strategy (i.e., seminal articles should be in the results of your search), as well as the quality and relevance of other studies included in the review.
Some published reviews can be found in many of the resources above, but not all reviews are published with Cochrane, JBI or the Campbell Collaboration. You will also need to search in databases such as CINAHL, ERIC, Medline, PsycINFO, and the Web of Science. Google scholar may also be useful when searching for published reviews.
There are many types of reviews, each suited to a particular purpose. The following need to be taken into consideration when deciding upon a review type1:
The following resources may help you decide which type of review best suits your research question(s):
References:
1. University of Alberta Library. (2024). Preparing for a comprehensive literature review. https://guides.library.ualberta.ca/reviewprep
Regardless of the type of review, a protocol is highly recommended for these reasons:
A protocol can be formal, informal, published, registered, or presented at conferences. Here are some typical components of a protocol:
Johanna Briggs Institute – Development of a scoping review protocol
Protocol: Bottrill, M., Cheng, S., Garside, R., Wongbusarakum, S., Roe, D., Holland, M. B., Edmond, J., & Turner, W. R. (2014). What are the impacts of nature conservation interventions on human well-being: A systematic map protocol. Environmental Evidence, 3(1), 16. https://doi.org/10.1186/2047-2382-3-16
Published article: McKinnon, M. C., Cheng, S. H., Dupre, S., Edmond, J., Garside, R., Glew, L., Holland, M. B., Levine, E., Masuda, Y. J., Miller, D. C., Oliveira, I., Revenaz, J., Roe, D., Shamer, S., Wilkie, D., Wongbusarakum, S., & Woodhouse, E. (2016). What are the effects of nature conservation on human well-being? A systematic map of empirical evidence from developing countries. Environmental Evidence, 5(1), 8. https://doi.org/10.1186/s13750-016-0058-7
Protocol: Michael Cusimano, Melissa Carpino, Madison Walker, Emily Rossi, Michael Tang, David Lightfoot, Robert Mann, Olli Saarela. The Effect of Recreational Cannabis Legalization on Substance Use, Mental Health, and Injury: A Systematic Review. PROSPERO 2024 Available from https://www.crd.york.ac.uk/PROSPERO/view/CRD42021265183
Published Article: Walker, M., Carpino, M., Lightfoot, D., Rossi, E., Tang, M., Mann, R., Saarela, O., & Cusimano, M. D. (2023). The effect of recreational cannabis legalization and commercialization on substance use, mental health, and injury: A systematic review. Public Health, 221, 87–96. https://doi.org/10.1016/j.puhe.2023.06.012
Protocol: Harfield, S., Davy, C., Kite, E., McArthur, A., Munn, Z., Brown, N., & Brown, A. (2015). Characteristics of Indigenous primary health care models of service delivery: A scoping review protocol. JBI Evidence Synthesis, 13(11), 43. https://doi.org/10.11124/jbisrir-2015-2474
Published article: Harfield, S. G., Davy, C., McArthur, A., Munn, Z., Brown, A., & Brown, N. (2018). Characteristics of Indigenous primary health care service delivery models: A systematic scoping review. Globalization and Health, 14(1), 12. https://doi.org/10.1186/s12992-018-0332-2
Knowledge syntheses require comprehensive searches across multiple databases. You need to search a minimum of two databases and typically, no more than five. Searching multiple databases is crucial for these reasons:
The databases you choose will depend on your research question and the disciplines in which relevant research may be conducted.
The databases you choose will also depend on what is available to you. Most databases are subscription based.
Use the library Research Guides or Find a Database list to select databases relevant to your topic, or consult with a librarian.
Commonly used databases included the following:
It is also recommended to set up individual accounts in each of the databases so that you can save your search strategies, edit them as needed, and rerun at later date, without having to enter the entire strategy again. See the tutorials below about how to to do this in select platforms.
Before you begin searching for articles, its important to plan for the documentation of your keyword brainstorming as well as your search strategies and results. You will need this information later for reporting and publication. You can use tables in a Word document or spreadsheets in Excel to do this.
Include information about:
Documenting your search results - Plymouth
Some databases have options to print or save the text of your search strategy to a Word, Rich Text, or PDF document. If a database doesn't have this option you can try copying and pasting the text or snipping an image of your strategy. To permanently save search strategies so that you can create an alert or run them again at a later date, you need to have an account in that database. See the tutorials below about how to to do this in platforms on which many of our databases are provided.
Search strategy tools
Consider using search filters (also called hedges). These are pre-constructed searches designed to search difficult concepts or concepts for which there are many search terms. They are usually database specific, but can be translated to work in different databses. Add them to a search strategy where appropriate and remember to cite them.
Canadian Agency for Drugs and Technologies in Health (CADTH)
Example filters:
Canada and Provinces - CINAHL database - University of Alberta
[mh Canada] or Canad* or "British Columbia" or "Colombie Britannique" or Alberta* or Saskatcehwan or Manitoba* or Ontario or Quebec or "Nouveau Brunswick" or "New Brunswick" or "Nova Scotia" or "Nouvelle Ecosse" or "Prince Edward island" or Newfoundland or Labrador or Nunavut or NWT or "Northwest Territories" or Yukon or Nunavik or Inuvialuit
Qualitative Studies - Medline (OVID) - University of Alberta
exp qualitative research/ or grounded theory/ or exp nursing methodology research/ or (qualitative or ethnol$ or ethnog$ or ethnonurs$ or emic or etic or hermeneutic$ or phenomenolog$ or lived experience$ or (Grounded adj5 theor$) or content analys$ or thematic analys$ or narrative analys$ or metasynthes$ or meta-synthes$ or metasummar$ or meta-summar$ or metastud$ or meta-stud$ or meta-ethnog$ or metaethnog$ or meta-narrat$ or metanarrat$ or meta-interpret$ or metainterpret$ or (qualitative adj5 meta-analy$) or (qualitative adj5 metaanaly$) or action research or photovoice or photo voice).mp.
Once you have drafted your search and before executing it in all databases, it's a good idea to review it for completeness, typos, correct use of search tools, correct combination of search lines, etc. A librarian can help with this or consult one of the following resources:
When creating thorough searches on certain topics, you might need to include some antiquated, non-standard, exclusionary, and possibly offensive terms to find older literature. The following article provides some guidance on how to address this:
Once you've completed a foundation search in one of the databases, it needs to be translated into all selected databases.
Important: Results of each database search need to be recorded in your documentation for later reporting.
Once you've completed the selection of articles for inclusion in your review, you may wish to add citation tracking (sometimes known as snowballing) to supplement your search. This entails checking those articles' reference lists and list of citing articles. Most databases have a Cited By/Citing Articles function, but Google Scholar is probably the best place to find citing articles. Here are some resources that may help with this process:
Sometimes you may need to update database searches because a significant amount of time has passed between its initial execution and manuscript preparation or you're updating a previously published review.
"Grey literature is information produced outside of traditional publishing and distribution channels, and can include reports, policy literature, working papers, newsletters, government documents, speeches, white papers, urban plans, and so on.
This information is often produced by organizations "on the ground" (such as government and inter-governmental agencies, non-governmental organisations, and industry) to store information and report on activities, either for their own use or wider sharing and distribution, and without the delays and restrictions of commercial and academic publishing. For that reason, grey literature can be more current than literature in scholarly journals."1
Examples of grey literature include:
References
1. Simon Fraser University Library. (2023). Grey Literature: What it is & how to find it. https://www.lib.sfu.ca/help/research-assistance/format-type/grey-literature
Important: Results of each database search need to be recorded in your documentation for later reporting.
Results of each database search need to be exported and saved for upload into a citation manager or review tool, for removing duplicates, screening and data extraction. Remember, screening doesn't happen in the databases.
Use the RIS file format when exporting results as it's the most compatible with a wide array of tools.
How you export articles varies between databases. Sometimes you need to set up an account in the database's platform (e.g., Proquest, Ebsco, OVID) in order to export a large number of citations. Below is a selection of library guides with instructions on how to do this. Most databases will also have instructions in their help sections.
Why do I need to remove duplicates?
If undetected, either could create bias in the conclusions of your review. Removing duplicates is an integral step in the process.
Where removing duplicate articles is concerned (i.e., matching articles vs duplicate use of data sets) some systematic review tools (e.g., Covidence and Rayyan) can automatically remove duplicate articles, but be aware they aren't perfect and will miss some. Screening tip: sorting the articles by title will help you identify missed duplicates. You can also remove duplicates with a citation management tool (e.g., Mendeley, Zotero, Endnote), which can give you more control over de-duplication.
Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016;104(3):240-3.
Important: No matter which deduplication approach you choose, don't delete duplicates and keep track of result totals before and after deduplication for later reporting. Some review tools will track this information.
Consider using a citation manager (e.g., Zotero, Mendeley, Endnote) to store and manage the records you find. It can assist you in organizing your records as you decide which ones to include in your review. Additionally, it is very effective for removing duplicate records. You can also link your own stored PDFs of your papers. Citation management software will also help you insert correctly formatted references into your final document and generate a reference list or bibliography.
These tools are specifically tailored to the needs of knowledge synthesis teams. In addition to reference management, some of these tools can also help with de-duplication, screening, data extraction, Prisma chart creation, analysis, track team progress, and facilitate communication between team members. Note that not every tool is appropriate for every kind of synthesis - be sure to choose the right fit for your project.
The following resources provide lists or comparison charts of core features for a variety of review tools:
Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018). Online tools supporting the conduct and reporting of systematic reviews and systematic maps: A case study on CADIMA and review of existing tools. Environmental Evidence, 7(1), 8. https://doi.org/10.1186/s13750-018-0115-5
Cowie, K., Rahmatullah, A., Hardy, N., Holub, K., & Kallmes, K. (2022). Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis. JMIR Medical Informatics, 10(5), e33219. https://doi.org/10.2196/33219
Screening involves a two-step process where you assess each article to determine if it meets the eligibility criteria outlined in your protocol, deciding whether it should be part of your review.
To minimize bias, it's recommended (and sometimes required by guidelines) to have at least two reviewers for screening—yourself and another member of your review team. In addition, you should ideally have a third person assigned the task of resolving conflicts.
Screening articles can be challenging, especially if you're new to this research methodology. It is recommended to pilot-test the screening process with reviewers to ensure the eligibility criteria are well defined and interpreted consistently. Reviewers should have a high minimum of 90% agreement among them in their screening process.
In their Screening for Articles guide, The University of Toronto provides excellent screening tips, as well as strategies to mitigate screening challenges.
Includes the review of titles and abstracts to remove obviously irrelevant material. Typically you're not required to provide reason(s) for exclusion in this step.
At this stage, you review the full-text to ensure it meets your eligibility criteria. You must provide reason(s) for exclusion in this step.
When you can't find the full-text, exclude the article and be sure to indicate this reason in your screening tool. When you set up your exclusion reasons in the tool, "Full-text unavailable" should be one of them.
See the list of Knowledge Synthesis Tools in the Manage Results section.
Quality Assessment, sometimes referred to as Critical Appraisal, is an integral part of some review methodologies. It involves the assessment of several related features, including but not limited to: risk of bias, quality of reporting, reproducibility, precision and external validity.
Articles
After the second screening stage, researchers need to read the full articles chosen for the review and pull out the important data using a form or template that has been designed to capture all of the detail relevant to the synthesis.
This form can be as detailed or simple as needed and can be coded for computer analysis if needed.
Guides and Resources
Pollock, D., Peters, M. D. J., Khalil, H., McInerney, P., Alexander, L., Tricco, A. C., Evans, C., de Moraes, É. B., Godfrey, C. M., Pieper, D., Saran, A., Stern, C., & Munn, Z. (2023). Recommendations for the extraction, analysis, and presentation of results in scoping reviews. JBI Evidence Synthesis, 21(3), 520–532. https://doi.org/10.11124/JBIES-22-001
Methods of analysis of data extracted varies between knowledge syntheses. Below are some resources that may help you with this process.
Tools for Analysis
Consult these guides to find information about reporting standards and tips on writing up your knowledge synthesis:
Writing the Report - Dalhousie University
Writing the Review - McGill University
How long does it take to complete a knowledge synthesis? The answer: it depends. Generally speaking, knowledge synthesis research is time intensive and often projects span a year or more to completion. Factors that effect the length of time it takes to do a knowledge synthesis:
The tables below provide some average timelines for different types of syntheses as well the stages in a knowledge synthesis project.
Try PredicTER, a tool for that calculates the time requirements for various tasks involved in reviewing evidence, from planning and coordination to quantitative synthesis and reporting.
| Type | Approximate number of months |
Why | Team |
|---|---|---|---|
| Literature review |
1-6 |
Bulk of time is spent reading resources & critically analyzing. Highly adaptable to resources & timelines. |
Typically one person |
| Mapping review | 6-12 |
Time is divided between search & visual depiction of the field. May require more time screening articles due to the larger volume of studies from covering a wider scope. |
Typically one person. Needs expertise in visualization tools. |
| Meta-analysis | 18-24 | Time is spent on all aspects of systematic review, as well as typically two stages of data analysis. | Experience in advanced statistical methods. |
| Rapid review | 1-8 |
Time is divided between designing question/ search strategy, reviewing and evaluating sources, and summarizing. Apply justified selection of methodology adjustments so as to meet resource limitations, especially time restrictions. |
Typically requires multiple people to reduce bias, depending on type of review |
| Scoping review | 2 - 24 | Time is spent formulating question, developing a protocol, refining search, screening studies, summarizing and synthesizing relevant studies, reporting results and identifying gaps and/or trends. |
Typical requires a minimum of two people. May require larger teams because of more search results to screen. May take longer than systematic review. |
| Systematic review | 12-24 | Time is spent defining research question, developing a protocol (inclusion/ exclusion criteria), testing, refining, and translating search strategies, title and abstract screening, full text screening, risk of bias assessment, data extraction & analysis. | Minimum of two persons to reduce bias. |
Table adapted from Evidence Synthesis, Library, University of Winnipeg.
| Month | Task | Description | Stage |
|---|---|---|---|
| 1-2 | Formulate research question | Refine and finalize your research question, ideally using a research question framework. | Preparation |
| 1-2 | Check for existing evidence synthesis reviews on your topic | Search for published evidence synthesis reviews or protocols that answer questions similar to yours | Preparation |
| 1-3 | Develop foundational search strategy | Retrieve and generate keywords and subject headings and incorporate appropriate syntax to create a search strategy which will find all relevant studies. You will need to translate your foundational search strategy to each database important to your topic. | Preparation |
| 2-3 | Write and register a protocol | Document the method, timeline and responsibilities for your review. Register the protocol to create an immutable record of your planned project and enhance accountability. | Preparation |
| 3-5 | Primary search | Execute your search strategy in each database | Collecting Data |
| 3-5 | Grey literature search | If aligned with your inclusion criteria, search for grey literature such as conference proceedings, reports, dissertations and theses. | Collecting Data |
| 4-5 | Export and de- duplicate citations | Export citations from your citation manager and remove duplicate citations | Collecting Data |
| 5-8 | Title and abstract screening | Screen titles and abstracts against your inclusion and exclusion criteria | Selecting & Synthesizing |
| 5-8 | Obtain full-text articles | Download from databases, request copies from inter-library loan, contact authors | Selecting & Synthesizing |
| 5-8 | Full-text screening | Exclude irrelevant articles based on inclusion/exclusion criteria | Selecting & Synthesizing |
| 9 | Supplemental search | Check reference lists of included articles for potential additional citations; new titles must go through screening process | Collecting Data/Selecting & Synthesizing |
| 9-11 | Data extraction | Extract all relevant data from each article and record for analysis | Selecting & Synthesizing |
| 9-11 | Data synthesis | If appropriate, convert extracted data to common measurements | Selecting & Synthesizing |
| 9-11 | Data analysis | Combine results from all included studies; analysis may be qualitative (thematic) or quantitative | Selecting & Synthesizing |
| 11 | Re-Run Foundational Search | Repeat the search to find new literature published since initial search. New articles must go through screening process | Data collection/Selecting & Synthesizing |
| 12 | Write report of review | Produce and disseminate final report of results | Report |
Table adapted from Knowledge Synthesis, Western Libraries.
Wollscheid, S., & Tripney, J. (2021). Rapid reviews as an emerging approach to evidence synthesis in education. London Review of Education, 19(1), Article 1. https://doi.org/10.14324/LRE.19.1.32
Rapid review method series - BMJ Evidence-Based Medicine