Towards Autonomous Multimodal LLMs: From Passive Perception to Proactive Understanding

The School of EECS is hosting the following HDR Progress Review 1 Confirmation Seminar:

Towards Autonomous Multimodal LLMs: From Passive Perception to Proactive Understanding

Speaker: Zhuojie Wu 

Abstract

Multimodal Large Language Models (MLLMs) are increasingly moving from offline content analysis towards continuous interaction with dynamic environments. However, current systems remain largely designed for bounded, prompt-driven inputs, making it difficult to support efficient, persistent, and adaptive understanding over continuously evolving streams.

This research investigates autonomous MLLMs that can continuously perceive multimodal information, maintain context over time, and respond based on the available evidence. To date, the research has focused on efficient visual perception and long-term streaming video understanding. Future research will explore adaptive online reasoning and the integration of evidence across multiple streams. The overall goal is to advance MLLMs from passive, one-shot perception towards efficient, stateful, and proactive understanding in dynamic real-world environments.

Bio

Zhuojie Wu is a  Ph.D. student at the School of EECS, University of Queensland, under the supervision of A/Prof Xin Yu, A/Prof Sen Wang, and Dr Heming Du. His research centres on efficient multimodal large language models.