AI-driven Approaches for More Effective Systematic Reviews

The School of EECS is hosting the following thesis review 3 seminar:

AI-driven Approaches for More Effective Systematic Reviews 

Speaker: Shuai Wang
Host: Prof Guido Zuccon

Abstract:

A systematic review (SR) is a rigorous and structured literature review process that synthesizes primary research studies to answer specific research questions. However, SRs are highly time-consuming and resource-intensive, often taking more than two years, requiring over 1,100 hours of effort, and costing more than $350K. This PhD research aims to reduce the workload and costs associated with SRs by improving the efficiency of the screening phase through three key research directions:

1. Enhancing Boolean Query Formulation: Low-quality Boolean queries often retrieve an excessive number of citations due to suboptimal formulation. This research explores two complementary approaches to improve query effectiveness: LLM-driven Automatic Boolean Query Formulation: This approach treats query formulation as a generative task, leveraging large language models (LLMs) to construct effective Boolean queries. Through extensive prompt engineering and evaluation of recent generative models, this work investigates how well LLMs can generate structured Boolean queries that balance recall and precision; MeSH Term Suggestion: In contrast, this approach formulates query improvement as a ranking task, utilizing neural BERT-based models to identify and rank relevant MeSH terms. The goal is to refine Boolean queries by integrating high-quality MeSH terms, enhancing retrieval effectiveness without overwhelming researchers with excessive citations.

2. Optimizing Screening: Screening is the most labour-intensive phase of SRs due to the low prevalence of relevant studies (average precision ~0.029). This work introduces two strategies to improve efficiency: Screening Prioritization: Methods are developed to rank retrieved citations using review titles, Boolean queries, and fine-tuned neural models, enabling earlier identification of relevant studies and reducing the overall screening burden; Automatic Screening Using Large Language Models: This research investigates the feasibility of using generative LLMs for automatic document classification, aiming to reduce the need for manual screening while maintaining reliability and transparency.

3. Exploiting Seed Studies: Seed studies, is a priori set of studies originally used to help systematic review creation, these studies can be leveraged to improve retrieval and ranking in SRs. This research first evaluates existing seed-driven methods using pseudo seed studies, then creates a new collection with real seed studies, and finally applies advanced seed-driven retrieval methods to improve the effectiveness of SR automation.

A key contribution of this research is the empirical validation of AI-driven automation methods in systematic reviews. The findings demonstrate that AI-driven query formulation, screening prioritization, and seed study exploitation can significantly reduce the screening burden while maintaining high recall. Additionally, this work provides new insights into the effectiveness of LLMs for Boolean query generation and document classification, offering a foundation for future research in systematic review automation.

Bio:

Shuai Wang is a Research Officer and PhD candidate at The University of Queensland, supervised by Professor Guido Zuccon, Associate Professor Bevan Koopman, and Dr. Harrisen Scells. His thesis focuses on AI-driven automation for systematic reviews, covering LLM-based Boolean query formulation, MeSH term suggestion, screening prioritization, automatic screening using LLMs, and seed study exploitation.

Beyond AI-driven techniques for systematic reviews, Shuai has contributed to general-domain IR tasks, including efficient retrieval models, federated search, ranking techniques, and retrieval-augmented generation (RAG). His recent work includes reproducing the 2D Matryoshka Model for IR, developing the Starbucks model, proposing novel resource selection methods for federated search, and exploring context compression in RAG. Additionally, he has investigated pseudo-relevance feedback (PRF) and rank fusion techniques to enhance retrieval performance.

Shuai’s research blends theory with large-scale empirical validation, and his work has been published in leading IR and NLP conferences, including SIGIR, ECIR, WSDM, WWW, and EMNLP.

More information: https://wshuai190.github.io/