Reliable Multimodal Recommender Systems

The School of EECS is hosting the following Progress Review 3 seminar:

Reliable Multimodal Recommender Systems

Speaker: Lijian Chen

Host: Prof. Hongzhi Yin

Abstract:

Multimodal recommender systems leverage rich item-side modalities (e.g., images and text) to improve recommendation quality, but this multimodality also expands the attack surface. For example, multimodal content is often provided by external parties (e.g., merchants) at scale and may contain human-imperceptible malicious signals. This thesis investigates the reliability of multimodal recommender systems through three research questions: (1) How reliable are images in visually-aware recommender systems, (2) How reliable is textual information in semantic-ID-based generative recommender systems, and (3) If both images and text are unreliable, how can we build a more reliable multimodal recommender system?

First, we study adversarial item promotion against visually-aware recommender systems and develop a guided diffusion attack that generates human-imperceptible adversarial images. The crafted images remain highly consistent with the originals while reliably manipulating ranking outcomes. Second, we reveal a new vulnerability of semantic ID-based recommenders by introducing SemBack, a backdoor item promotion attack that poisons item metadata to make RQ-VAE produce adversarial semantic IDs, while preserving objective facts and maintaining imperceptibility. Finally, to improve reliability beyond item-side modalities, we explore user-side modalities and propose a multi-agents based user values mining framework grounded in Schwartz’s Theory of Basic Values, and integrate user-value semantics into user embeddings via contrastive learning, yielding promising gains in recommendation performance and robustness.

Bio:

Lijian Chen is a Ph.D. student at the School of EECS at the University of Queensland, under the supervision of Prof. Hongzhi Yin and A/Prof. Rocky Chen. He completed his B.Eng. in ICT Engineering at the University of Technology Sydney in 2019 and his M.InfTech. in Software Development and Data Analytics at the University of Technology Sydney in 2021. His research interests include trustworthy recommender systems and multimodal recommender systems.

About Data Science Seminar

This seminar series is hosted by EECS Data Science.

Venue

Room: 78-421 or Zoom Link: https://uqz.zoom.us/j/88918969788