Decentralized On-device Machine Learning and Unlearning

The School of ITEE is hosting the following thesis review seminar:

Decentralized On-device Machine Learning and Unlearning

Speaker: Guanhua Ye
Host: A/Prof. Hongzhi Yin

Abstract: The widespread use of mobile devices and Internet of Things (IoT) devices has resulted in an enormous amount of data being generated by these devices. However, collecting and centralizing this data raises significant privacy concerns, which hinder the development of machine learning models that could leverage this data. To address these concerns, a new collaboration paradigm is proposed - decentralized learning, which involves client raw data sealing and peer-to-peer communication. The absence of a central (cloud) server not only removes the dependency on the trustworthiness of a network dominator but also optimizes the communication overhead and personalization of client models. In recent years, much effort has been dedicated to improving the performance and versatility of decentralized learning, yet a comprehensive study from infrastructure to paradigm is still in demand.

This thesis systematically studies the problem of decentralized learning across three layers: the device layer, the network layer, and the user layer. The device layer focuses on on-device models, the network layer focuses on inter-device communications, and the user layer focuses on the interactions between personal information and aggregative knowledge. The challenges identified in these layers include the energy and convergence efficiency of on-device models (device layer), the collaboration between structurally heterogeneous models (network layer), and being forgotten in decentralized collaboration (user layer). This thesis proposes four solutions to tackle the aforementioned challenges, leading to the following key contributions: (i) designed a frequency extraction model with a one-to-multi structure, achieving high energy efficiency due to its compatibility with low-duty-cycle sensors; (ii) expanded the Block Coordinate Descent optimization (BCD) algorithm into decentralized scenarios to optimize convergence performance; (iii) proposed a novel communication strategy called messenger distillation for supporting high-quality heterogeneous collaboration; (iv) proposed a collaboration paradigm based on seed model ensemble that facilitates fast exact machine unlearning. Overall, this thesis identified the intrinsic gap between theoretical studies and real-world applications with respect to the decentralized learning problem and provides practical bottom-to-top solutions.

Speaker Biography: Mr. Guanhua Ye obtained his B.E. in Biomedical and Information Engineering from Northeastern University in 2019. Then he started his Ph.D. in computer science under the supervision of A/Prof. Hongzhi Yin and Dr. Miao Xu. His research interests include decentralized learning, heterogeneous collaboration, and machine unlearning.