A new benchmark called norregel has been developed to test language models' ability to extract dated numeric values from Norwegian legal texts. The benchmark, which includes 916 items extracted from Lovdata's open data, assesses whether models provide correct, outdated, wrong, or uninterpretable answers based on the effective date of the information. In initial tests, Opus 5.5 and Sonnet 4.6 showed varying accuracy, with Opus 5.5 correctly answering 46.7% of primary items. Subsequent preregistered runs indicated that outdated answers are more frequent for values that took effect after a model's knowledge cutoff, though the robustness of this finding requires further investigation. AI
IMPACT This benchmark could drive improvements in LLM accuracy for time-sensitive legal and financial data extraction.
RANK_REASON The item describes a new academic benchmark for evaluating language models on a specific task, which falls under research. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
