Characterising and Improving the Reliability of Low-bit Quantization in Foundation Models

The School of EECS is hosting the following PhD Progress Review 1 Confirmation Seminar:

Characterising and Improving the Reliability of Low-bit Quantization in Foundation Models

Speaker: Venkataramanan Shivaraman (V Shivaraman)

Host: Prof. Mahsa Baktashmotlagh

Abstract: Low-bit quantization reduces model storage and can accelerate inference, enabling efficient deployment of modern foundation models. However, these efficiency gains must be assessed alongside the reliability of quantized models under deployment conditions. Strong performance on clean benchmarks does not establish robustness when the distribution changes. BitRot investigates this issue through a systematic study of post-training quantization in Vision Transformers across quantization methods, model configurations, and visual distribution shifts. We find that four-bit models can retain much of their clean ImageNet accuracy while suffering substantially larger losses under corruptions, artistic renditions, and naturally adversarial examples. Mixed calibration provides modest and inconsistent recovery in the tested settings. We further develop a post-hoc model to characterise the interaction between quantization and distribution shift, supported by evidence across configurations and individual predictions. These findings highlight the importance of explicit robustness evaluation and provide a foundation for understanding and improving the reliability of low-bit models.

Bio: V Shivaraman is a PhD student in the school of EECS at UQ, supervised by Prof. Mahsa Baktashmotlagh and Prof. Siamak Layeghy.