Efficient Visual Learning via Compression of Representations, Models and Datasets

The School of EECS is hosting the following PhD Progress Review 3 Thesis Seminar:

Efficient Visual Learning via Compression of Representations, Models and Datasets

Speaker: Yudong Chen
Host: Prof Xue Li

Abstract: Deep model-based methods have become increasingly important in computer vision because of their superior feature learning ability. However, utilizing these models in the real world is still challenging due to the high computation complexity and large memory costs. Firstly, with the exponential increase of data, the traditional real-valued representation vectors generated by the deep models are computationally expensive for similarity measurement. Secondly, the deep models usually require GPUs to achieve real-time inference, which is infeasible on some edge devices (e.g., mobile phones). Lastly, optimizing a deep model on a device with limited computing power and memory resources is time-consuming due to the usage of large-scale training sets. The lack of efficiency largely restricts the deployment of deep models in practical applications. Therefore, it is critical to develop improved methods for efficient visual learning.

This thesis proposes a series of compression approaches to reduce the computation complexity and memory usage of deep models for efficiency improvement. Specifically, there are three goals we aim to achieve, including the compression of representations, the compression of deep models, and the compression of datasets. The contributions of this thesis are as follows: (i) to accelerate the similarity computation between representation vectors produced by the deep models, a binary code learning method is proposed to quantify the vectors with similarity preservation in the original real-valued feature space; (ii) to reduce the inference costs of deep models, a simple but effective projector ensemble-based feature distillation method is designed to transfer the knowledge from a cumbersome teacher network to a lightweight student network; (iii) in addition to leveraging knowledge from the teacher's features, a novel teacher weights-aware distillation method is also developed to further improve the feature learning ability of the student network; and (iv) to improve the training efficiency of deep models with knowledge distillation, a sample selection strategy is proposed to eliminate the less important samples on the dataset and obtain fast optimization of deep models. In the end, this thesis discusses the potential improvement of the proposed methods and the future research directions for efficient visual learning.

Speaker bio: Yudong Chen is a PhD candidate at the School of Electrical Engineering and Computer Science (EECS), The University of Queensland, under the supervision of AsPr Sen Wang and Dr Jiajun Liu. He received his master's degree from Shenzhen University. His research interests include hashing learning and network compression.