A new benchmark called IBBench-Light has been developed to evaluate how well language models respond to external directives, specifically when those directives involve applying a procedure or reading text from an external record. The benchmark uses twelve semantic bases to create 144 paired responses per model, with a focus on paired exact-contract accuracy (PECA), which requires both parts of a response pair to meet their specified conditions. Initial tests show that models like Qwen perform inconsistently, with significant differences between average performance and complete paired success rates. The study also highlights the impact of technical configurations, such as end-of-sequence token sets, on model performance. AI
IMPACT This benchmark could lead to more robust LLM evaluations, particularly for models designed to interact with external information sources.
RANK_REASON The item is an academic paper detailing a new benchmark for evaluating language models. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- CatalyzeX
- Connected Papers
- CORE Recommender
- DagsHub
- Gotit.pub
- Hugging Face
- IBBench-Light
- Litmaps
- Phi Llm
- Qwen
- ScienceCast
- scite Smart Citations
- SmolLM2
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
