TREC AutoJudge: A Meta-Evaluation of Retrieval-Augmented Generation Evaluation Methodologies

The Data Science Discipline of the School of EECS is hosting the following guest seminar:

TREC AutoJudge: A Meta-Evaluation of Retrieval-Augmented Generation Evaluation Methodologies

Speaker: Dr Maik Fröbe, Bauhaus-Universität Weimar, Germany

Abstract:

Retrieval-Augmented Generation (RAG) systems are increasingly being deployed in a wide range of information retrieval applications. While the offline evaluation of traditional information retrieval systems is well established (e.g., the community has decades of experience with the Cranfield paradigm), the offline evaluation of RAG systems is still in an exploratory stage, with many evaluation methodologies currently being investigated. A systematic comparison of these RAG evaluation methodologies, potentially spanning many years, can help identify which methodologies are suitable for different scenarios. In this context, the upcoming TREC AutoJudge shared task is a meta-evaluation that compares RAG evaluation methodologies across many scenarios. The talk will introduce the TREC AutoJudge shared task and the research artefacts it will produce (e.g., relevance judgments, nugget banks, LLM caches, software, etc.). The talk will discuss the types of comparisons these resources enable and conclude with an open discussion to brainstorm research directions and collaborations.

Bio:

Maik Fröbe studied Computer Science at Leipzig University (M.Sc.), Martin Luther University Halle-Wittenberg, and Friedrich Schiller University Jena (PhD). Since July 2026, he has been a Postdoctoral Researcher in the Webis Group at Bauhaus-Universität Weimar. His current research interests include information retrieval, LLM-as-a-Judge, retrieval-augmented generation, and reproducibility.

Maik Fröbe is visiting the DLab this week after attending SIGIR in Melbourne.