Towards Cost-Efficient Federated Multi-Agent Reinforcement Learning with Learnable Aggregation

The School of EECS is hosting the following PhD progress review 1 seminar

Towards Cost-Efficient Federated Multi-Agent Reinforcement Learning with Learnable Aggregation

Speaker: Yi Zhang
Host: Dr Sen Wang

Abstract: Inspired by the principle that "the collective wisdom is greater than the sum of its parts", cooperative Multi-Agent Reinforcement Learning (MARL) demonstrates invaluable potential in tackling intricate multi-robot tasks within dynamic environments. However, some current methodologies in this field prove to be a double-edged sword. While they can simplify the training of MARL algorithms, yielding impressive performance in game scenarios, the widely used parameter-sharing setting can infringe upon the privacy of learning robots and inhibit the scaling of algorithms to realistic environments. Although part of the research has moved towards non-parameter sharing with the aim to enhance agents’ performance, critical factors related to efficiency, such as communication and computation overheads, are largely ignored. Addressing this, we introduce a client-server framework designed to fortify MARL in real-world scenarios, achieving scalability, parallelization, and privacy preservation. Our approach, named Cost-Efficient Federated Multi-Agent Reinforcement Learning with Learnable Aggregation (FMRL-LA), decouples MARL training, enabling collective learning to simultaneously optimize both system performance and efficiency. This is accomplished by leveraging centralized critics to measure agents’ utility during global model updates on the server side. Theoretically, we account for the Markovian randomness in agents’ experiences, establishing convergence bounds for our method and proposing a general utility function for analyzing MARL in realistic environments. To comprehensively validate our method, we extend a challenging multi-agent autonomous driving environment to the federated learning paradigm. We evaluate our FMRL-LA against competitive MARL baselines, and the experimental results show that our approach skillfully strikes a balance between performance and efficiency considerations.

Speaker Biography: Mr Yi Zhang is a PhD student from the Data Science group at the School of Electrical Engineering and Computer Science, the University of Queensland (UQ), Australia. He received his bachelor’s degree from Peking University. He is currently working towards his PhD degree under the supervision of Dr. Sen Wang and Dr. Jiajun Liu. His research interests include Multi-agent Reinforcement Learning and Federated Learning.