M-PFT method improves calibration for vision-language models

Researchers have introduced M-PFT, a new method for prompt-tuning vision-language models (VLMs) that addresses calibration issues. Prompt-tuned VLMs often exhibit underconfidence on familiar classes and overconfidence on new ones. M-PFT utilizes an angular-margin approach to enhance class discriminability by minimizing cosine similarity between class embeddings and aligning tunable embeddings with their zero-shot counterparts. Experiments across various methods and datasets demonstrate M-PFT's effectiveness in reducing calibration error on novel classes while maintaining performance on base classes and accuracy, with minimal computational cost. AI

IMPACT This research offers a novel approach to improve the reliability and calibration of vision-language models, potentially enhancing their performance in real-world applications.

RANK_REASON The cluster describes a new research paper detailing a novel method for prompt-tuning vision-language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• cosine similarity
• embedding
• M-PFT
• Prompt Tuning by Context Template Pool Optimisation for Vision-Language Model
• vision-language model
• Vision--Language Models

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →