Towards Robust Multimodal Foundation Models against Poisoning Threats

 

Seminar Title: Towards Robust Multimodal Foundation Models against Poisoning Threats

 

Speaker: Zhifang Zhang

 

Host: Ruihong Qiu

 

Abstract:

Multimodal foundation models learn powerful capabilities from image-text data, but poisoned training samples can implant backdoors that induce malicious behaviour while preserving performance on clean inputs. This confirmation proposal summarises three studies from my first year that investigate these threats, progressing from multimodal classification to large vision-language models. Repulsive Visual Prompt Tuning (RVPT) suppresses classification backdoors by adapting visual representations using few-shot clean data. CleanSight identifies abnormal cross-modal attention during language generation and selectively removes suspicious visual tokens at test time without retraining. TokenSwap exposes a more evasive threat: poisoned models can generate fluent responses that name the correct objects while reversing their semantic roles. Together, these studies connect the analysis of backdoor mechanisms with efficient defenses and the evaluation of semantic corruption. Future research will broaden this investigation to text-to-image and text-to-video generation and deepen it from individual models to agents interacting with real-world environments rich in diverse modalities. The agent studies will examine poisoning through external environments and unreviewed harnesses, and how memory compression may preserve or amplify its influence.

 

Bio:

Zhifang Zhang is a PhD student at the School of Electrical Engineering and Computer Science, The University of Queensland, supervised by Dr Miao Xu. His research focuses on the security and robustness of multimodal foundation models, with an emphasis on poisoning threats and defense mechanisms.