Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark
A new benchmark, PiSAR, has been developed to evaluate screen-conditioned action prediction in AI models. The benchmark revealed that a fine-tuned Qwen3-VL-8B-Instruct model significantly outperformed frontier zero-shot models like Claude Opus 4.7 and GPT-5.5, achieving a semantic similarity score of 0.783 compared to the frontier models' scores around 0.46-0.48. This suggests that while large, frontier models are powerful, specialized fine-tuning can yield substantial improvements on specific tasks. The study also noted a potential mismatch between the fine-tuning recipe and the Gemma-4-26B-A4B-IT model, indicating that model architecture and training methodology are crucial for effective fine-tuning. AI
IMPACT Demonstrates the significant performance gains achievable through fine-tuning on specific tasks, potentially guiding future model development and application strategies.