Multimodal Recommender Systems: Threats and Countermeasures in the Generative AI Era

The School of EECS is hosting the following PhD Progress Review 1 Seminar.

Multimodal Recommender Systems: Threats and Countermeasures in the Generative AI Era

Speaker: Lijian Chen
Host: Prof. Hongzhi Yin

Abstract: In the era of burgeoning data, recommender systems play a crucial role in offering personalized suggestions across diverse industries. These systems traditionally predict user preferences by learning latent features from extensive collaborative data, like user-item interactions. However, a persistent challenge is the data sparsity issue due to the limited user-item interaction information, making it hard to accurately capture user interests. To conquer the data sparsity problem, one classical solution is to incorporate auxiliary information to assist recommender system predict user preference, namely multi-modal recommender systems.  Among these auxiliary information, visual signals play a significant role as user behavior is heavily influenced by visual tastes in many scenarios, such as fashion, food, and micro-video recommendations. Therefore, visually-aware recommender systems have been widely applied and researched. While visual information enhances recommender system’s performance, this crucial information is usually provided by external sources due to the large number of items. We argue that this setting may leave a backdoor for adversaries to manipulate the visually-aware recommender system by creating and uploading adversarial images. This research introduces a corresponding attack method to uncover the actual vulnerabilities within these systems. Specifically, we have developed an adversarial attack named the Item Promotion by Diffusion Generated Image (IPDGI) attack, which is crafted with consideration for real-world conditions and constraints. IPDGI aims to generate adversarial images to highlight security concerns associated with using images provided by third parties through the trusted channel of the system. Extensive experiments have been conducted on two real-world recommendation datasets with three visually-aware recommender systems. The experimental results showcase the effectiveness and inconspicuous of our IPDGI attack, revealing the severe threats caused by untrust visual signals.

Bio: Lijian Chen is a Ph.D. student at the School of EECS at the University of Queensland, under the supervision of Prof. Hongzhi Yin and Dr. Rocky Chen. He completed his B.Eng. in ICT Engineering at the University of Technology Sydney in 2019 and his M.InfTech. in Software Development and Data Analytics at the University of Technology Sydney in 2021. His research interests include trustworthy recommender systems and multimodal recommender systems.