Multimodal Perception, Retrieval, and Reasoning for Context-Aware Systems

The Data Science Discipline of the School of Electrical Engineering and Computer Science invites you to:

 

PhD Confirmation Seminar Title: Multimodal Perception, Retrieval, and Reasoning for Context-Aware Systems

 

Speaker: Honghao Fu

Host: Dr. Ruihong Qiu

 

Abstract:

Multimodal reasoning systems increasingly operate over heterogeneous, noisy, and structurally complex observations. However, in many cases, only a subset of the available information may be relevant to a target task, while redundant or irrelevant observations may interfere with downstream reasoning. One challenge is therefore context construction: how to distill complex multimodal observations into clean, compact, and coherent evidence to better support downstream reasoning. This confirmation seminar summarizes the speaker's published research work over the past year on this problem, comprising three studies in embodied decision-making (VistaWise), multimodal in-context learning (ContextNav), and long-video understanding (VideoStir). Specifically, VistaWise grounds task-conditioned visual observations in a dynamic vision–text knowledge graph and retrieves dependency-aware context for agent planning and action in Minecraft. ContextNav treats multimodal in-context learning as an adaptive contextualization problem, retrieving, filtering, and structurally aligning contextual demonstrations to construct effective contexts for task adaptation. VideoStir addresses long-video understanding by representing video contexts as spatio-temporal graphs, retrieving relationally connected clips, and selecting fine-grained visual evidence according to the reasoning intent of the query. Across these different settings, the findings suggest that downstream reasoning outcomes depend not only on the capability of the reasoning model, but also on the quality of the context constructed around it. Preserving task-relevant structure, reducing irrelevant or inconsistent information, and aligning evidence selection with reasoning needs can improve downstream performance without modifying the main reasoning model. Furthermore, this seminar outlines potential directions for further research during the remainder of the speaker's candidature.

 

Biography:

Honghao Fu is a PhD student at the School of Electrical Engineering and Computer Science, The University of Queensland, supervised by Dr. Yujun Cai and Dr. Miao Xu. He received his MSc from the School of Electrical and Electronic Engineering at Nanyang Technological University, Singapore. Prior to this, he obtained his BEng from Southeast University, China. His research interests lie in large language models, agentic systems, and multimodal understanding.