AI-driven Approaches for Effective Systematic Review Literature Screening

The School of EECS is hosting the following PhD Progress Review 3 Seminar

Speaker: Xinyu Mao

Host: Dr Joel Mackenzie

Abstract: 

A systematic review (SR) is a type of literature review that appraises and syntheses primary studies as evidence, in answering specific research questions. However, conducting a SR is time-consuming and labour-intensive, typically spans more than 2 years to complete with over 1,100 hours of human effort and costs more than $350K. Most of the costs arises in the screening phase, where human reviewers assess thousands of studies for relevance—despite an average inclusion rate of only 2.9%. This thesis aims to reduce the workload and costs of SR creation during the screening phase through two main research directions:

1. Neural approaches for Screening Prioritisation. To reduce the time and effort on irrelevant studies, prioritising relevant studies is a common strategy used in traditional machine learning-based solutions. In this direction, we investigate how neural models, such as BERT, can improve screening effectiveness over traditional methods. We first examine BERT-based models in the traditional screening workflow, then apply dense retrieval for higher effectiveness and efficiency in screening. We also build open-source screening tools powered by our dense-retrieval-based solution and large language models (LLMs) for an improved screening experience.

2. Boolean Query Performance Prediction for Effective Screening. Boolean queries are used to retrieve candidate studies, but poorly formulated queries can lead to excessive irrelevant studies or missed key ones. This direction explores the use of LLMs to predict query quality by estimating its correlation with the precision and recall of the retrieved studies, enabling early identification of low-performing queries and reducing unnecessary screening effort.

This thesis contributes to SR screening by providing effective and efficient AI-driven solutions from two aspects: screening method and query for retrieval. Our findings show that the screening burden can be further reduced with our AI-driven screening prioritisation and query performance prediction, which are made accessible via open-source tools. This thesis also offers insights into the feasibility of using LLMs to automate the screening phase and to enhance the quality of screening, providing a basis for future study in SR automation.

Bio:
Xinyu Mao is a PhD candidate from the School of EECS at the University of Queensland, supervised by Prof. Guido Zuccon, A/Prof. Bevan Koopman, and Dr. Harrisen Scells. Prior to his PhD, he received a Master’s degree in Data Science from the University of Queensland in 2022. His research focuses on Information Retrieval and Systematic Review Automation. He has also open-sourced screening tools for effective systematic review screening, including DenseReviewer.