A new audit of Chinese Legal Case Retrieval (LCR) benchmarks reveals that the primary factor driving performance is not advanced retrieval models, but rather the way benchmarks are constructed around legal charges. Researchers found that simply ranking cases by their shared primary charge, combined with basic BM25 ranking, recovers nearly all of the performance gap between simple and complex systems on the LeCaRDv2 benchmark. This suggests that current benchmarks may not accurately measure true legal reasoning ability, as relevance is often determined by the charge itself rather than deeper case analysis. The study proposes a reusable protocol to screen future benchmarks for similar construct-validity issues. AI
RANK_REASON Academic paper detailing a new audit and proposed protocol for evaluating AI benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]
Read on arXiv cs.IR (Information Retrieval) →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
