A new benchmark called InsufficiencyBench has been developed to evaluate the ability of legal AI systems to handle underspecified user queries. The benchmark, which includes 202 items annotated by practicing attorneys across six legal domains and 24 US jurisdictions, focuses on whether models can identify missing legally material information, determine what is missing, and avoid premature conclusions. Evaluations of ten frontier models revealed that none exceeded an F2 score of 0.46 for missing-element identification, with a median recall of 0.44, indicating that current models either hedge excessively or make unfounded assumptions. AI
IMPACT Highlights critical limitations in current legal AI, suggesting a need for models that can better handle ambiguity and identify missing information before providing advice.
RANK_REASON The cluster contains a research paper introducing a new benchmark for evaluating AI systems. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
