Comparing the performance of narrow vs. broad search strategies when using machine learning-based software for title/abstract screening
DOI:
https://doi.org/10.5195/jmla.2026.2286Keywords:
AI, machine learning, evidence synthesis as topic, systematic review as topic, Screening Tools, search strategy developmentAbstract
Objective: To retrospectively evaluate workload implications and recall performance of narrower or broader database search strategies when using active learning screening tools.
Methods: A convenience sample of 10 completed reviews was used to assess search strategy performance in ASReview LAB, an open-source systematic review software tool. For each review, a single database search strategy was selected and then revised to either broaden (n = 9) or narrow (n = 1) the scope. Results from both the more sensitive (broader) and more precise (narrower) search strategies were labeled as relevant or irrelevant based on inclusion in the completed review. The labeled result sets were uploaded into the ASReview LAB simulation module, which mimics the process of human screening. Metrics such as number of records screened to reach true recall of 95% or more were recorded. The effects of three different stopping rules on workload and recall were also explored.
Results: For quantitative systematic reviews, the difference in absolute screening time required to reach 95% recall between broader or narrower search strategies was minimal (≤35 minutes). In contrast, for qualitative systematic reviews and other review types, broader search strategies led to increased workload. With respect to stopping rules, the time-based stopping heuristic resulted in substantial workload increases when broader search strategies were employed.
Conclusions: Time savings achieved through the use of semi-automated screening tools may not always offset additional screening time required by broader, more sensitive search strategies. Librarians and information specialists should consider a variety of factors when determining the appropriate balance between search sensitivity and specificity in the context of semi-automated screening tools.
References
1. Nussbaumer-Streit B, Ellen M, Klerings I, Sfetcu R, Riva N, Mahmić-Kaknjo M, Poulentzas G, Martinez P, Baladia E, Ziganshina LE, Marqués ME, Aguilar L, Kassianos AP, Frampton G, Silva AG, Affengruber L, Spjker R, Thomas J, Berg RC, Kontogiani M, Sousa M, Kontogiorgis C, Gartlehner G. Resource use during systematic review production varies widely: a scoping review. J Clin Epidemiol. 2021 Nov;139:287–96.
2. Haddaway NR, Westgate MJ. Predicting the time needed for environmental systematic reviews and systematic maps. Conserv Biol. 2019 Apr;33(2):434–43.
3. Affengruber L, Van Der Maten MM, Spiero I, Nussbaumer-Streit B, Mahmić-Kaknjo M, Ellen ME, Goossen K, Kantorova L, Hooft L, Riva N, Poulentzas G, Lalagkas PN, Silva AG, Sassano M, Sfetcu R, Marqués ME, Friessova T, Baladia E, Pezzullo AM, Martinez P, Gartlehner G, Spijker R. An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review. BMC Med Res Methodol. 2024 Sep 18;24(1):210.
4. Tóth B, Berek L, Gulácsi L, Péntek M, Zrubka Z. Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Syst Rev. 2024 Jul 8;13(1):174.
5. Marshall IJ, Noel‐Storr A, Kuiper J, Thomas J, Wallace BC. Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner’s guide. Res Synth Methods. 2018 Dec;9(4):602–14.
6. Van De Schoot R. Active learning explained. ASReview Blog. April 12, 2022. Accessed November 3, 2025. https://asreview.nl/blog/active-learning-explained/
7. Plunkett C. Machine learning – the game changer for trustworthy evidence. Covidence Blog. N.d. Accessed November 3, 2025. https://www.covidence.org/blog/machine-learning-the-game-changer-for-trustworthy-evidence/.
8. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010 Dec;11(1):55.
9. Adam GP, Davies M, George J, Caputo E, Htun JM, Coppola EL, Holmer H, Kuhn E, Wethington H, Ivlev I, Balk EM, Trikalinos TA. Machine Learning Tools To (Semi-) Automate Evidence Synthesis [Internet]. Agency for Healthcare Research and Quality (AHRQ); 2025 Jan [cited 2025 Feb 25]. Available from: https://effectivehealthcare.ahrq.gov/products/machine-learning-tools/white-paper.
10. Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Metzendorf MI, Noel-Storr A, Paynter R, Rader T, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies [last updated March 2025]. In: Cochrane Handbook for Systematic Reviews of Interventions version 651. Cochrane. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04.
11. O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev. 2019 Dec;8(1):143.
12. Sousa-Pinto B, Marques-Cruz M, Neumann I, Chi Y, Nowak AJ, Reinap M, Awad M, Nothacker M, Trucl M, Brozek J, Alonso-Coello P, Wiercioch W, Qaseem A, Akl EA, Schünemann HJ, 2024 Board of Trustees of the Guidelines International Network, Munn Z, El-Ansary L, Kopp I, Langendam M, James R, Alam M. Guidelines International Network: Principles for Use of Artificial Intelligence in the Health Guideline Enterprise. Ann Intern Med. 2025 Jan 28;ANNALS-24-02338.
13. Affengruber L, Nussbaumer-Streit B, Hamel C, Van Der Maten M, Thomas J, Mavergames C, Spijker R, Gartlehner G. Rapid review methods series: Guidance on the use of supportive software. BMJ Evid-Based Med. 2024 Aug;29(4):264–71.
14. Ross-White A, Lieggi M, Palacio F, Solomons T, Swab M, Rothfus M, Takahashi J, Cardoso D. (2024). 2.4 Search methodology for JBI Evidence Syntheses. In: JBI manual for evidence synthesis. https://doi.org/10.46658/JBIMES-24-01
15. Bramer WM, De Jonge GB, Rethlefsen ML, Mast F, Kleijnen J. A systematic approach to searching: an efficient and complete method to develop literature searches. J Med Libr Assoc [Internet]. 2018 Oct 4 [cited 2025 Mar 24];106(4). Available from: http://jmla.pitt.edu/ojs/jmla/article/view/283.
16. McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80.
17. Van De Schoot R, De Bruin J, Schram R, Zahedi P, De Boer J, Weijdema F, Kramer B, Huijts M, Hoogerwerf M, Ferdinands G, Harkema A, Willemsen J, Ma Y, Fang Q, Hindriks S, Tummers L, Oberski DL. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021 Feb 1;3(2):125–33.
18. Shemilt I, Simon A, Hollands GJ, Marteau TM, Ogilvie D, O’Mara‐Eves A, Kelly MP, Thomas J. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Synth Methods. 2014 Mar;5(1):31–49.
19. Hamel C, Hersi M, Kelly SE, Tricco AC, Straus S, Wells G, Pham B, Hutton B. Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses. BMC Med Res Methodol. 2021 Dec;21(1):285.
20. ASReview LAB developers. ASReview Lab - A tool for AI-assisted systematic reviews [Internet]. Utrecht University; Available from: https://doi.org/10.5281/zenodo.3345592.
21. Donnan JR, Grandy CA, Chibrikov E, Marra CA, Aubrey-Bassler K, Johnston K, Swab M, Hache J, Curnew D, Nguyen H, Gamble JM. Comparative safety of the sodium glucose co-transporter 2 (SGLT2) inhibitors: a systematic review and meta-analysis. BMJ open. 2019 Jan 1;9(1):e022577.
22. Donnan JR, Rash J, Drakes DH, Swab M, Jacques Q, King A, Linkiewich D, Mcintyre V. (2025). "Peer support in chronic pain management: a patient co-led systematic review." Late breaking abstract at Canadian Pain Society Annual Scientific Meeting. Toronto ON, 1-4 May.
23. Donnan J, Shogan O, Bishop L, Swab M, Najafizada M. Characteristics that influence purchase choice for cannabis products: a systematic review. Journal of cannabis research. 2022 Feb 1;4(1):9.24.
24. Fawcett JM, Baldwin MM, Whitridge JW, Swab M, Malayang K, Hiscock B, Drakes DH, Willoughby HV. Production improves recognition and reduces intrusions in between-subject designs: An updated meta-analysis. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale. 2023 Mar;77(1):35.
25. Smith-Young J, Pike A, Swab M, Chafe R. Parents’ and guardians’ experiences of barriers and facilitators in accessing autism spectrum disorder diagnostic services for their children: a qualitative systematic review. JBI evidence synthesis. 2025 Jan 1;23(1):6-8.
26. Small SP, de Boer C, Swab M. Barriers to and facilitators of labor market engagement for individuals with chronic physical illnesses in their experiences with work disability policy: a qualitative systematic review. JBI evidence synthesis. 2022 Feb 1;20(2):348-536.
27. Balan M, Bodley T, Heybati K, Deng J, Gaetani M, Maratta C, Swab M, Burns K, Murthy S. Environmental sustainability evidence in intensive care medicine: a scoping review [under review].
28. Pacheco L, Mercerat C, Aunos M, Cousineau MM, Goulden A, Swab M, Brenton B, Moyo S. Uncovering Reproductive Injustice Toward Women With Disabilities. International Perspectives in Psychology. 2024 Jul 1.
29. Young C, Patey C, Norman P, Chan T, Hurley O, Swab M, Asghari S. Identifying relevant topics and training methods for emergency department flow training. Canadian Journal of Emergency Medicine. 2022 Dec;24(8):837-43.
30. Hillis C, Bhattacharjee M, Al Mousawi B, Martens R, Eltanahy R, Ono S, Hui M, Pham B, Swab M, Cormack G, Grossman M, Bagheri E, Marshall Z. AI ethics education: a scoping review of pedagogy, curriculum, and assessment [under review].
31. Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes. BMC Med Res Methodol. 2020 Dec;20(1):256.
32. Wang Z, Nayfeh T, Tetzlaff J, O’Blenis P, Murad MH. Error rates of human reviewers during abstract screening in systematic reviews. Bencharit S, editor. PLOS ONE. 2020 Jan 14;15(1):e0227742.
33. Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev. 2016 Dec;5(1):140.
34. Callaghan MW, Müller-Hansen F. Statistical stopping criteria for automated screening in systematic reviews. Syst Rev. 2020 Dec;9(1):273.
35. Ros R, Bjarnason E, Runeson P. A Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies. In: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering [Internet]. Karlskrona Sweden: ACM; 2017 [cited 2025 May 7]. p. 118–27. Available from: https://dl.acm.org/doi/10.1145/3084226.3084243.
36. König L, Zitzmann S, Fütterer T, Campos DG, Scherer R, Hecht M. An evaluation of the performance of stopping rules in AI ‐aided screening for psychological meta‐analytical research. Res Synth Methods. 2024 Nov;15(6):1120–46.
37. Boetje J, Van De Schoot R. The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses. Syst Rev. 2024 Mar 1;13(1):81.
38. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016 Dec;5(1):210.
39. Burns JK, Etherington C, Cheng‐Boivin O, Boet S. Using an artificial intelligence tool can be as accurate as human assessors in level one screening for a systematic review. Health Inf Libr J. 2024 Jun;41(2):136–48.
40. Chai KEK, Lines RLJ, Gucciardi DF, Ng L. Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews. Syst Rev. 2021 Dec;10(1):93.
41. Gartlehner G, Wagner G, Lux L, Affengruber L, Dobrescu A, Kaminski-Hartenthaler A, Viswanathan M. Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study. Syst Rev. 2019 Dec;8(1):277.
42. Gates A, Gates M, Sebastianski M, Guitard S, Elliott SA, Hartling L. The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews. BMC Med Res Methodol. 2020 Dec;20(1):139.
43. Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, Hartling L. Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Syst Rev. 2019 Dec;8(1):278.
44. Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Syst Rev. 2018 Dec;7(1):45.
45. Howard BE, Phillips J, Miller K, Tandon A, Mav D, Shah MR, Holmgren S, Pelch KE, Walker V, Rooney AA, Macleod M, Shah RR, Thayer K. SWIFT-Review: a text-mining workbench for systematic review. Syst Rev. 2016 Dec;5(1):87.
46. Howard BE, Phillips J, Tandon A, Maharana A, Elmore R, Mav D, Sedykh A, Thayer K, Merrick BA, Walker V, Rooney A, Shah RR. SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation. Environ Int. 2020 May;138:105623.
47. Przybyła P, Brockmeier AJ, Kontonatsios G, Le Pogam M, McNaught J, Von Elm E, Nolan K, Ananiadou S. Prioritising references for systematic reviews with RobotAnalyst: A user study. Res Synth Methods. 2018 Sep;9(3):470–88.
48. Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015 Dec;4(1):80.
49. Tsou AY, Treadwell JR, Erinoff E, Schoelles K. Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer. Syst Rev. 2020 Dec;9(1):73.
50. Van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, Van Der Palen J, Doggen CJM, Lenferink A. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open. 2023 Jul;13(7):e072254.
51. Byrne F, Hofstee L, Teijema J, De Bruin J, Van De Schoot R. Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study. Syst Rev. 2024 Jul 8;13(1):175.
52. Ferdinands G, Schram R, De Bruin J, Bagheri A, Oberski DL, Tummers L, Teijema JJ, Van De Schoot R. Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records. Syst Rev. 2023 Jun 20;12(1):100.
53. Teijema JJ, Hofstee L, Brouwer M, De Bruin J, Ferdinands G, De Boer J, Vizan P, Van Den Brand S, Bockting C, Van De Schoot R, Bagheri A. Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders. Front Res Metr Anal [Internet]. 2023 May 16 [cited 2025 Jul 10];8. Available from: https://www.frontiersin.org/articles/10.3389/frma.2023.1178181/full.
54. Campos DG, Fütterer T, Gfrörer T, Lavelle-Hill R, Murayama K, König L, Hecht M, Zitzmann S, Scherer R. Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research. Educ Psychol Rev [Internet]. 2024 Mar [cited 2025 Jul 10];36(1). Available from: https://link.springer.com/10.1007/s10648-024-09862-5.
55. Giummarra MJ, Lau G, Gabbe BJ. Evaluation of text mining to reduce screening workload for injury-focused systematic reviews. Inj Prev. 2020 Feb;26(1):55–60.
56. Oude Wolcherink MJ, Pouwels XGLV, Van Dijk SHB, Doggen CJM, Koffijberg H. Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles? Expert Rev Pharmacoecon Outcomes Res. 2023 Oct 21;23(9):1049–56.
57. Pijls BG. Machine Learning assisted systematic reviewing in orthopaedics. J Orthop. 2024 Feb;48:103–6.
58. Robledo S, Grisales Aguirre AM, Hughes M, Eggers F. “Hasta la vista, baby” – will machine learning terminate human literature reviews in entrepreneurship? J Small Bus Manag. 2023 May 4;61(3):1314–43.
59. Harmsen W, De Groot J, Harkema A, Van Dusseldorp I, De Bruin J, Van Den Brand S, Van De Schoot R. Machine learning to optimize literature screening in medical guideline development. Syst Rev. 2024 Jul 11;13(1):177.
60. Rogers K, Hagerman L, Neil-Sztramko S, Dobbins M. Learning on the job: using Artificial Intelligence to support rapid review methods. J Med Libr Assoc. 2024 May 22;112(2):148–9.
61. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005 Nov 5;331(7524):1064–5.
Downloads
Additional Files
Published
Versions
- 2026-04-13 (2)
- 2026-04-13 (1)
Issue
Section
License
Copyright (c) 2025 Michelle Swab
This work is licensed under a Creative Commons Attribution 4.0 International License.
