A new evaluation framework called Just Keep Prompting (JKP) has been developed to assess the stability of Vision-Language Models (VLMs) during extended conversations. The framework uses strategies like adversarial negation and Socratic interrogation to challenge models over multiple turns. Initial tests on GPT-4o, Gemini 2.5 Pro, and Qwen3-VL-30B revealed that while overall accuracy may not change drastically, models exhibit significant instability, with correct answers regressing and incorrect ones sometimes recovering. The study found that repeated prompting can act as a destabilizer rather than a reasoning aid, with performance varying significantly across different models. AI
IMPACT Highlights critical VLM limitations in conversational stability, suggesting current evaluations may not capture real-world performance under pressure.
RANK_REASON Academic paper detailing a new evaluation methodology for VLMs. [lever_c_demoted from research: ic=1 ai=1.0]
- Adversarial Negation
- Context-Aware Socratic Summarization
- Gemini 2.5 Pro
- GPT-4o
- Just Keep Prompting
- Pure Socratic Interrogation
- Qwen3-VL-30B
- STAR benchmark
- Vision--Language Models
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
