Evaluating automated or artificial intelligence search tools for evidence synthesis

Abstract

To advance information retrieval science for producing evidence syntheses at Canada’s Drug Agency, the Research Information Services team developed a replicable process to evaluate automated or artificial intelligence (AI) search tools. The team inventoried 51 tools in the fall of 2023 and built a flexible evaluation instrument to inform adoption decisions and enable comparison between tools. Building on this foundational evaluation work, the team further conducted a comparative analysis on three top-ranked tools in the fall of 2024. The investigation confirmed that these automated or AI tools have inconsistent and variable performance for the range of information retrieval tasks performed by Information Specialists at Canada’s Drug Agency. Implementation recommendations from this study informed a “fit for purpose” approach where Information Specialists leverage automated or AI search tools for specific tasks or project types.