What Generators Want from Search: Aligning Retrieval with the Needs of Generative AI

The School of EECS is hosting the following HDR Progress Review 1 Confirmation Seminar:

What Generators Want from Search: Aligning Retrieval with the Needs of Generative AI

Speaker: Jonathan Ross

Host: Prof Guido Zuccon

Abstract: Search has traditionally been defined as the interaction between a user and an information retrieval (IR) system to satisfy an information need. From the outset, IR evaluation has assumed that system utility lies in serving human users, and that this utility can be approximated by measuring. Consequently, retrieval models and evaluation measures were designed around human-centred notions of relevance, and search systems have been optimised accordingly.

These same search systems are now central to Generative AI, particularly in Retrieval-Augmented Generation (RAG), where a retriever supplements the input of a large language model (generator) with external documents. RAG has enabled state-of-the-art performance on knowledge-intensive tasks, improved attribution of answers to sources, and allowed models to incorporate new knowledge without retraining.

Yet, retrieval in RAG still relies on methods optimised for humans rather than for generators. Human-oriented practices such as ranking documents by semantic relevance reflect assumptions about how people search: that they read results top to bottom, expect the most relevant item first, and stop once their need is satisfied. These assumptions are embedded in evaluation measures such as Normalised Discounted Cumulative Gain (NDCG), which reward systems for placing relevant documents higher in the list. Yet empirical studies reveal that such ranked lists can impair generators. A model may prefer relevant documents at the end of the list or even perform better when non-relevant documents appear first. These findings raise a deeper question: what principles should guide the design of retrievers for generators?

This thesis advances a framework for retrieval explicitly aligned with the needs of generative models. It first situates the research within the history of human-centred IR and motivates the shift toward generator-aligned evaluation. It then critically examines how assumptions embedded in classical retrieval theory and metrics diverge from generator behaviour. Finally, it presents empirical studies that quantify these divergences and inform the design of retrievers optimised for generator utility. Together, these contributions move beyond traditional relevance to define the principles of retrieval for generative AI.

Bio: Mr Jonathan Ross is a PhD candidate from the Data Science group at the School of Electrical Engineering and Computer Science, the University of Queensland (UQ), Australia. He received his Bachelor’s degree in Mechanical Engineering from the United States Naval Academy and a Master’s of Data Science from UQ. His research focuses on Retrieval Augmented Generation (RAG), under the supervision of Dr. Anton van der Vegt, Associate Professor Bevan Koopman, and Professor Guido Zuccon.

About Data Science Seminar

This seminar series is hosted by EECS Data Science.

Venue

Room: 78-421
Zoom: https://uqz.zoom.us/j/2211955903?omn=85488414677