New LEGIT dataset evaluates LLM legal reasoning with issue tree rubrics

Researchers have developed LEGIT, a new dataset containing 24,000 legal reasoning instances designed to evaluate the quality of LLM-generated legal arguments. This dataset converts court judgments into hierarchical trees of arguments and conclusions, serving as rubrics for assessing reasoning traces. Experiments using LEGIT indicate that LLMs' legal reasoning is significantly impacted by issue coverage and correctness, and that retrieval-augmented generation (RAG) and reinforcement learning (RL) offer complementary benefits, with RAG enhancing overall capability and RL improving correctness at the cost of coverage. AI

IMPACT Introduces a new dataset and evaluation framework for assessing LLM legal reasoning capabilities, potentially improving the reliability of AI in legal applications.

RANK_REASON This is a research paper introducing a new dataset and evaluation methodology for LLM legal reasoning.

Read on arXiv cs.CL →

• arXiv
• LEGIT
• LLM
• RL

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →