New AI benchmark tests policy reasoning, Qwen3 14B fine-tune achieves human-level performance

Researchers have developed a new benchmark called Dual-Axis Policy Entailment (DAPE) to evaluate how well AI models can reason about changes in government policies. The benchmark, which uses expert-annotated data from Chinese government policies, assesses both whether policies pertain to the same domain and how they relate to each other (continue, broaden, narrow, or reverse). Initial testing showed that leading frontier API models performed poorly, but a fine-tuned open-weight model, Qwen3 14B, achieved human-level agreement on the task. AI

IMPACT This benchmark could lead to more robust AI systems capable of understanding and analyzing complex policy shifts, crucial for applications in legal tech, public administration, and geopolitical analysis.

RANK_REASON The cluster describes a new academic benchmark and evaluation of AI models on policy reasoning, published on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• China
• Cohen's kappa
• Dual-Axis Policy Entailment
• Hugging Face
• Qwen3 14B

• paper
• policy
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →