2-1 Searching and Reviewing the Literature

Overview

An important first step in a scholarly project is to “do a literature review”. Why are literature reviews important? What are the different types of literature reviews? Could literature reviews become stand-alone scholarly projects? In this chapter, we will explore the answers to these questions.

Vignette

Darla and Lemy stared at their screen for the second hour in a row. They had been tasked to conduct a literature review but they weren’t even really sure how to get started. Their supervisor had tasked them to conduct a proper literature review prior to starting their study protocol, and now they were in the library starting at the blank search bar on the PubMed website.

Darla broke the silence first: “Lemy… I think we probably need to understand WHY we’re doing this literature review first… I mean, are we just trying to find papers that help us understand the topic? Are we trying to find papers that have used similar methods to our current study design? I’m not sure what we should even search for?”

They were very confused. As someone who had not done a lot of research before, Lemy felt very lost as well.

“I think we need help… Do you think we could go ask one of the librarians?” Lemy asked.

“Great idea,” replied Darla. “I wonder if someone at the help desk could book us in…”

“Oooh, and I think Dr. Martinez also sent us a link to the HPER manual where we might be able to get a sense of what we need to do…”

Deeper Dive into this Concept

Why do we need to do a literature review prior to starting your project?

There are many important reasons to do a literature review. Contributing to research is akin to joining a conversation. We need to be familiar with the conversation before contributing to it in a meaningful manner. This helps us understand where the gaps are, so we could endeavour to fill them (1,2). Once a gap and a research question is identified, literature review can also bring us up-to-speed on what is already known in the subject, so we can build off of the work already done. Missing relevant literature is one of the top ten reasons for rejection of manuscripts, considering that the reviewers are likely authors and experts on the subject (3). Ultimately, the goal of scholarly projects is to advance the field. With limited resources, it is extremely important to identify the scope and nature of the research subject in order to do so efficiently.

When does the literature review help you as an author of a project?

A great literature review can be the foundation of your expertise in a whole domain of research. When conducting literature reviews, you will find that it can help you gain a greater grasp of what has come before and how you might add to the discourse within an area. (1,4) Engaging with a thorough reading of the literature in your area of interest will help you begin your journey to become a scholar and expert in a domain. Let’s walk through how an investigator might approach the literature review for a brand new domain (e.g. new area of interest for your thesis).

BEFORE

Before you begin your study, literature can help you in a number of the following ways:

1. Identify what thoughts and ideas have existed in the field before;
2. Highlight new research questions pondered previously (e.g. scientists expressing what they think are future directions);
3. Help you to find key thinkers and scientists within the field. (Pro Tip: Set Google Scholar Alerts if you can for these folks so that if they publish something new in this domain you’ll find out right away).

DESIGN

When formulating your study, it is important to pay attention to what has come before within your field. Sometimes this will be so that you can build upon the lines of reasoning from previous studies, but sometimes it will be to foreshadow a gap you have noticed in the approaches that have come before. For instance, in a certain domain you may find that mainly researchers have only used experimental designs to date. Some will want to build upon those experiments to continue the lines of thought around these prior works – possibly conducting more nuanced experiments. Others may feel that since the field is populated by experimentalists, it may be worthwhile to bring new epistemologies and methods into this domain. Either way you wish to proceed, a very well conducted literature review is the key to knowing what has come before and to provide you insights into the types of research used to understand your domain.

WRITING

A great literature review will be very helpful when writing your manuscript as well. First off, most authors do experience some sort of writer’s block at some point. If you conduct a methodological literature review, which is not topic driven, but rather revolves around the type of study design that you intended to complete you will have greater success.

Often when you are embarking on writing up a study protocol you may find yourself stuck trying to find the right phrases and ideas for how to describe your intended study design. A methodology-based review will help you to gain a better sense of the diverse ways in which authors communicate their studies using the same methods you intend to use. This will empower you to better understand key components of study design. Some would even suggest hitting up resources like the EQUATOR network (5) to see if there are reporting guidelines for the type of study you are hoping to complete – these can scaffold you to think through your study protocol or, later, your manuscript. After all, if you do not factor elements into your study up front, you will have no possibility of reporting this later.

With regards to the usual domain-specific literature review, reading avidly around your topic will help you to construct and contextualize your problem or observation, the gap that you have identified in the literature, and further bolster the case that you have the right question in the end. Citing the right papers can help you to really build your case in the introduction and/or background of your paper. Then, the literature review comes in handy again later as it will help you contextualize your findings in the discussion. Usually in the discussion, authors will begin by summarizing key findings from their study – but then they will be expected to compare and contrast their findings to the field writ large. Having a good literature review that provides you with a wide basis of knowledge about the field will be key to success in formulating a great discussion for your paper. Lingard describes papers as having a number of different storylines: 1) Coming full circle, 2) Deep exploration, and 3) Surprise insight (6). Each of these story formats for a manuscript usually come to fruition during the discussion. The literature that you foreshadow in the introduction, then, often re-emerge within the discussion and now take the spotlight. Without an adequate grounding in the literature that surrounds your paper, it is virtually impossible to write a great introduction or discussion.

PUBLISHING

Finally, a very pragmatic reason for engaging in a great literature review is that your reviewers and editors will expect this of your manuscript. A great literature review helps you survive peer review (3) and get beyond editorial desk rejection (7). In his 2001 paper, Georges Bordage highlights that one of the topic reasons reviewers reject manuscripts is often cited as having an “…incomplete, inaccurate, or outdated review of the literature” (3). In a more recent paper, Meyer and colleagues (7) found that desk rejections (i.e. when editors don’t even send a paper out for review) often occurred when there is a “weak discussion and/or conclusions” or “inadequate or incomplete introduction”. A great literature review can help to remedy these key problems that often occur.

Can you turn literature reviews into education scholarship?

The simple answer to this question is: Yes, but you don’t always have to do this. Many authors do like to engage in multiple wins; many scholars find taking that literature review you have to do anyways and turning it into a work of scholarship is a way to fulfil this mandate. In fact, many scholars have made careers on publishing rigorously conducted reviews, and as such, it is certainly possible to capitalize on turning your literature review into a work of education scholarship.

That said, the rigour of your literature review will vary widely when you are conducting various formats of literature review. Often when you are merely doing your literature review for your own purposes, you’ll hit a search engine like Google Scholar or PubMed and just start entering keywords to hopefully find papers that are relevant. Perhaps you’ll use tools like Web of Science to trace through papers and their citations to find other relevant papers. Though circuitous – these methods can be highly helpful to you when you’re just learning about a topic and reading for edification.

When you decide, however, that you would like to turn your literature review into a proper piece of academic scholarship, then it is important to conduct a thorough and systematic search where you find the right phrasing and use the exact keywords from the indices of choice. As such, for a scholarly literature review where you aim to publish, you will likely want to involve an academic librarian to peer review your search strategy prior to conducting the search, and moreover, you will need to be more meticulous in your completion of that search. One need only review the relevant reporting guidelines for various types of reviews [systematic review (PRISMA (8), and the PRISMA-S extension (9), meta-analysis (MOOSE (10), or scoping review (PRISMA-ScR (11)] to see that there are high expectations for the types of detailed records you must keep throughout your formal search in order to ensure your literature review is publishable.

What are best practices or reporting guidelines for conducting these reviews?

The following table (2.1.1) lists various types of reviews that are described in the literature (12,13) as well as their matched reporting guideline an example from Health Professions Education.

The Nuts and Bolts of a Literature Review

Depending on the type of review you are conducting, then the level or rigour that you will need to have for a literature review will vary greatly. If you are just trying to get a sense of the conversations in a field, many scholars increasingly use technologies like Google Scholar, which has some advantages over keyword-driven databases like PubMed, EMBASE, PsychInfo or ERIC. Google Scholar uses a more inclusive search and does not restrict its search just to title and keywords – it searches the entire text of a paper. Google Scholar can be advantageous as then you may be able to discover synonyms or overlapping terminology that you may not yet know about.

Conducting a Pilot Search

Sometimes when you are starting on a literature review, you might not know what keywords to use at the start. A pilot search can be helpful to help you complete this task. You can use broader scholarly search engines like Google Scholar to find papers (and grey literature) of relevance, and then you can review what keywords or title language is used by the authors to describe the paper. This pilot search can be used to start constructing a list of possible keywords that you may wish to use in your search strategy.

The Role of a Librarian

Though a librarian is not always necessary for a formal review team, they are an invaluable resource and may be very helpful to you in the early stages of your search process. Academic librarians at some institutions also hold professorial roles, and therefore may wish to be included as co-authors in the paper – so be attuned to your own colleagues and discuss with them early whether this is a goal of theirs. Some librarian staff members may not seek involvement in the authorship process and simply wish to be thanked. It is always good practice, however, to have an open discussion about authorship with any substantive member of the team – and in the case of a formal, publishable literature review paper, your librarian colleague may serve a similar role to a methodologist on another study. Make sure you extend the courtesy and opportunity for them to earn their seat at the table.

The Role of the Crowd or Experts

Sometimes when you are super strapped and lost, then there may be a role for polling the crowd or consulting experts in the field. One group has used a social media-driven strategy to consult the #MedTwitter and #MedEd communities consistently throughout several of their review papers (27-29), allowing authors to gain a broader lens on the topic and find papers that are related but may not have the same keywords.

Expert consultation with those who are well-versed in a field may also be quite useful to discern related papers that may not be on your radar because of a number of factors. True experts in a domain may be aware of literature that is in another related field (e.g. K-12 education, organizational psychology) where they have worked previously or have colleagues who conduct related research.

Reporting Guidelines: A Novice Scholar’s Best Friend when Reading and Writing the Literature

Reporting guidelines, as we have mentioned earlier in this chapter, can be very helpful to novice scholars. They provide an overview of the reporting standards that experts have agreed upon that different types of studies should adhere to when being published in the literature. While authors may not adhere to every reporting standard (as you will find when you begin to write your own manuscripts), it can be helpful for you

When reading the literature…

Remember, it is very important to remain skeptical whenever reading and citing literature. In order to do this, scholars have created reporting standards which help fields to understand the rigour with which authors are expected to be explicit about their scholarly processes. You can find most of the reporting standards in the EQUATOR network website (5). These guidelines can help to scaffold your review of a specific type of study – for instance, you might find that the PRISMA guidelines (8) are useful when reviewing a systematic review or meta-analysis, or STROBE guidelines (30) are useful for guiding readers to look for important aspects of observational studies. That said, these reporting guidelines do not replace the importance of critically appraising the work for relevance (e.g. does the study examine a similar phenomenon and/or population as you?) and meaningfulness (e.g. are the results meaningful to you as a reader/educator?).

When conducting a review…

While this research primer does not address the specific protocols on how to do a full review, we would also suggest that the reporting guidelines for various types of reviews can be helpful to you as a scaffold for what you should take care to include in your first draft. If you are doing a formal review, then here are some questions (related to elements in the PRISMA-S extension on literature searches) that may set you off on the right path:

1. Do you list all the database names?
2. Did you use a multi-database searching software? (e.g. OVID)
3. Did you search any study registries?
4. Did you review any online or print source purposively to examine or triangulate your literature? (e.g. manual review of journal tables of contents, proceedings of conferences, websites)
5. Did you conduct a citation search? (e.g. reviewing the cited references in the papers you found originally)
6. Did you contact any further authors or experts?
7. Did you include your full search strategy (copied and pasted exactly as you ran it)?
8. Did you have any limits (e.g. date, time, language) to your search? If you did, please justify or explain why you did so?
9. Did you filter your search somehow? If so, please explain the filter process.
10. Did you reuse a search strategy previously described in the literature? If so, have you cited this previous source and then have you explained how you modified things since the last time the search strategy was used (e.g. changed the date parameters)?
11. Did you include the date of the last time you ran your search strategy?
12. If a substantive amount of time has passed since you ran your original search, did you find a way to update the search? (e.g. rerun the search with the time since you began the study, set email alerts for certain authors or relevant keywords?)
13. Did your search strategy undergo peer review by an expert? (e.g. a librarian)
14. Did you document your total records originally pulled from each database or other sources?
15. Did you have a process to reconcile duplicates across the databases or searched sources?

Vignette Conclusion

A few days later Darla and Lemy logged into a video conference with one of the academic librarians. They had been sent a chapter from something called the Health Professions Education Research (HPER) Primer that had helped them to clarify what they wanted to do next.

The librarian smiled at them and asked: “How can I help?”

“Well… we were wondering if you could help us with constructing a robust search strategy on our topic. We’ve done some pilot searching in Google Scholar, and we’ve found some keywords from relevant papers that we think will help us do the job in PubMed, but we’re having trouble with converting things to other search engines…”

“Wow,” the librarian stated, “You two really have a good start on this. Let’s see what you’ve got?”

References

1. Lingard L, Watling C. Problem/Gap/Hook Introductions. InStory, Not Study: 30 Brief Lessons to Inspire Health Researchers as Writers 2021 (pp. 7-14). Springer. https://link.springer.com/content/pdf/10.1007/978-3-030-71363-8.pdf 
2. Lingard L. Joining a conversation: the problem/gap/hook heuristic. Perspectives on Medical education. 2015 Oct;4(5):252-3. https://link.springer.com/article/10.1007/s40037-015-0211-y
3. Bordage G. Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. Academic medicine. 2001 Sep 1;76(9):889-96. https://journals.lww.com/academicmedicine/fulltext/2001/09000/reasons_reviewers_reject_and_accept_manuscripts_.10.aspx4.
4. Bordage G, Dawson B. Experimental study design and grant writing in eight steps and 28 questions. Medical education. 2003 Apr;37(4):376-85. https://onlinelibrary.wiley.com/doi/full/10.1046/j.1365-2923.2003.01468.x
5. EQUATOR NETWORK – Enhancing the QUAlity and Transparency of health Research. Accessed last October 5, 2021. Available at: https://www.equator-network.org/https://www.equator-network.org/
6. Lingard L. Does your discussion realize its potential?. Perspectives on medical education. 2017 Oct;6(5):344-6. https://link.springer.com/article/10.1007/s40037-017-0377-6
7. Meyer HS, Durning SJ, Sklar DP, Maggio LA. Making the first cut: an analysis of academic medicine editors’ reasons for not sending manuscripts out for external peer review. Academic Medicine. 2018 Mar 1;93(3):464-70. https://journals.lww.com/academicmedicine/Fulltext/2018/03000/Making_the_First_Cut__An_Analysis_of_Academic.37.aspx
8. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLOS Medicine 2021;18(3):e1003583. doi: https://doi.org/10.1371/journal.pmed.1003583
9. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Systematic reviews. 2021 Dec;10(1):1-9. https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-020-01542-z
10. Brooke BS, Schwartz TA, Pawlik TM. MOOSE Reporting Guidelines for Meta-analyses of Observational Studies. JAMA surgery. 2021 Apr 7. https://jamanetwork.com/journals/jamasurgery/article-abstract/2778476
11. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MD, Horsley T, Weeks L, Hempel S. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of internal medicine. 2018 Oct 2;169(7):467-73. https://www.acpjournals.org/doi/full/10.7326/M18-0850
12. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health information & libraries journal. 2009 Jun;26(2):91-108. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1471-1842.2009.00848.x
13. Murnaghan A, Weersink K, Thoma B, Hall AK, Chan T. The writer’s guide to education scholarship in emergency medicine: systematic reviews and the scholarship of integration (part 4). Canadian Journal of Emergency Medicine. 2018 Jul;20(4):626-33. https://www.cambridge.org/core/journals/canadian-journal-of-emergency-medicine/article/writers-guide-to-education-scholarship-in-emergency-medicine-systematic-reviews-and-the-scholarship-of-integration-part-4/A232626870BE1C00046989886EF3B84F
14. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology. 2012 Dec;12(1):1-8. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-12-181
15. Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015 Dec 1;24(4):230-5. https://www.tandfonline.com/doi/abs/10.1179/2047480615Z.000000000329
16. Norman G, Dore K, Grierson L. The minimal relationship between simulation fidelity and transfer of learning. Medical education. 2012 Jul;46(7):636-47. https://onlinelibrary.wiley.com/doi/full/10.1111/j.1365-2923.2012.04243.x
17. Kahlke RM, McConnell MM, Wisener KM, Eva KW. The disconnect between knowing and doing in health professions education and practice. Advances in Health Sciences Education. 2020 Mar;25(1):227-40. https://link.springer.com/article/10.1007/s10459-019-09886-5
18. Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC medicine. 2013 Dec;11(1):1-4. https://link.springer.com/article/10.1186/1741-7015-11-21
19. Price T, Wong G, Withers L, Wanner A, Cleland J, Gale T, Prescott‐Clements L, Archer J, Bryce M, Brennan N. Optimising the delivery of remediation programmes for doctors: A realist review. Medical Education. 2021 Mar 26. https://onlinelibrary.wiley.com/doi/full/10.1111/medu.14528
20. Gottlieb M, Chan TM, Zaver F, Ellaway R. Confidence‐competence alignment and the role of self‐confidence in medical education: A conceptual review. Medical Education. 2021 Jun 27. https://onlinelibrary.wiley.com/doi/full/10.1111/medu.14592
21. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International journal of social research methodology. 2005 Feb 1;8(1):19-32. https://www.tandfonline.com/doi/full/10.1080/1364557032000119616
22. Colquhoun HL, Levac D, O’Brien KK, Straus S, Tricco AC, Perrier L, Kastner M, Moher D. Scoping reviews: time for clarity in definition, methods, and reporting. Journal of clinical epidemiology. 2014 Dec 1;67(12):1291-4. https://www.sciencedirect.com/science/article/pii/S0895435614002108
23. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implementation science. 2010 Dec;5(1):1-9. https://link.springer.com/article/10.1186/1748-5908-5-69
24. Chan TM, Dzara K, Dimeo SP, Bhalerao A, Maggio LA. Social media in knowledge translation and education for physicians and trainees: a scoping review. Perspectives on medical education. 2020 Feb;9(1):20-30. https://link.springer.com/article/10.1007/s40037-019-00542-7
25. Cheng A, Eppich W, Grant V, Sherbino J, Zendejas B, Cook DA. Debriefing for technology‐enhanced simulation: a systematic review and meta‐analysis. Medical Education. 2014 Jul;48(7):657-66. https://onlinelibrary.wiley.com/doi/full/10.1111/medu.12432
26. Ilgen JS, Sherbino J, Cook DA. Technology‐enhanced simulation in emergency medicine: a systematic review and meta‐analysis. Academic Emergency Medicine. 2013 Feb;20(2):117-27. https://onlinelibrary.wiley.com/doi/full/10.1111/acem.12076
27. Thoma B, Gottlieb M, Boysen-Osborn M, King A, Quinn A, Krzyzaniak S, Pineda N, Yarris LM, Chan T. Curated collections for educators: five key papers about program evaluation. Cureus. 2017 May;9(5). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5453746/
28. Quinn A, Chan TM, Sampson C, Grossman C, Butts C, Casey J, Caretta-Weyer H, Gottlieb M. Curated collections for educators: five key papers on evaluating digital scholarship. Cureus. 2018 Jan;10(1). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5837232/
29. Schnapp BH, Alvarez AA, Bianchi R, Caretta‐Weyer H, Jewell C, Kalantari A, Lee E, Miller D, Quinn A. Curated collection for clinician educators: Six key papers on residency recruitment. AEM Education and Training. 2021 Apr;5(2):e10597. https://onlinelibrary.wiley.com/doi/full/10.1002/aet2.10597
30. Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. The Lancet. 2007 Oct 20;370(9596):1453-7. https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(07)61602-X/fulltext

About the Authors

name: Catherine Tong

institution: McMaster University

Catherine Tong (she/her) is an associate Clinical Professor affiliated with the Dept of Family Medicine at McMaster University.  She is the Director of the Royal College Clinician Educator Diploma AFC (Area of Focused Competence) program at the McMaster site. She also leads the Faculty Development program at the Waterloo Regional Campus, as well as the Inspired Teaching Team of the Program for Faculty Development at the Faculty for Health Sciences at McMaster University.

name: Teresa Chan

institution: McMaster University

Teresa Chan (she/her) is an Associate Professor in the Department of Medicine within the Division of Emergency Medicine and cross-appointed to the Division of Education & Innovation. She is an associate member of the Department of Health research methods, Evidence, and Impact. She is also the Associate Dean, Continuing Professional Development Office (CPD) and a Clinician Scientist, McMaster Education Research, Innovation & Theory (MERIT) Program, Faculty of Health Sciences, McMaster University.