Distilling from Vision-Language Models for Improved OOD Generalization in Vision Tasks

In the world of artificial intelligence, Vision-Language Models (VLMs) such as CLIP have made a significant impact due to their impressive generalization across a variety of data distributions. However, the high costs associated with training these models and curating their data have resulted in a vendor-client paradigm, where a vendor trains a large-scale VLM and clients gain only input-output access on a pay-per-query basis in a black-box setting.

The research paper "Distilling from Vision-Language Models for Improved OOD Generalization in Vision Tasks" by Sravanti Addepalli et al. addresses this challenge by proposing a new approach called Vision-Language to Vision-Align, Distill, Predict (VL2V-ADiP). This method aims to maximize the inference cost-efficiency by distilling the VLM to a student model using the limited available task-specific data.

The VL2V-ADiP method aligns the vision and language modalities of the teacher model with the vision modality of a pre-trained student model. Then, the aligned student features are refined using the VLM's image and text embeddings. The distillation process incorporates the rich representations of the VLM image encoder and the superior generalization of the text embeddings, thereby maintaining the balance between the cost of inference and the quality of the distilled student model.

The authors report that this approach significantly improves the In-Domain (ID) accuracy and out-of-distribution (OOD) generalization. The paper shares that the proposed approach achieves state-of-the-art results on standard Domain Generalization benchmarks in a black-box teacher setting, and also when weights of the VLM are accessible.

This research opens up new avenues for maximizing the utility of large-scale VLMs in a cost-effective manner, while also ensuring superior generalization capabilities. The implications of this research could be far-reaching, particularly for organizations looking to leverage the power of VLMs without incurring prohibitive costs.

Read the whole article here: http://arxiv.org/abs/2310.08255v1