Fine-tuned Qwen3-VL-8B-Instruct outperforms Claude Opus 4.7, GPT-5.5 on PiSAR benchmark

A new research paper introduces the PiSAR benchmark for evaluating screen-conditioned action prediction. The study found that a fine-tuned Qwen3-VL-8B-Instruct model significantly outperformed frontier zero-shot models like Claude Opus 4.7 and GPT-5.5 on this benchmark, achieving a semantic similarity score of 0.783 compared to the frontier models' scores around 0.45-0.48. However, the same fine-tuning approach applied to Gemma-4-26B-A4B-IT yielded much lower scores, suggesting a mismatch between the model architecture and the fine-tuning recipe. AI

IMPACT Demonstrates the significant impact of fine-tuning on specific tasks, potentially guiding future model development and evaluation strategies.

RANK_REASON The cluster contains a research paper detailing a new benchmark and model performance evaluations. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Claude Opus 4.7
• Gemma-4-26B-A4B-IT
• GPT-5.5
• Qwen3-VL-8B-Instruct

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →