A new benchmark study reveals that large language models (LLMs) exhibit a significant disparity in their ability to accurately cite legal documents, particularly when dealing with different jurisdictions. While models like Gemini 2.5 Flash, GPT-OSS-120B, and Nemotron-3-Super-120B perform well on the EU's General Data Protection Regulation (GDPR), they frequently fabricate citations for the Saudi Personal Data Protection Law (PDPL). This fabrication occurs regardless of the query language (Arabic or English) and is often presented with high confidence, highlighting a critical need for verification safeguards before relying on LLMs for legal compliance. AI
IMPACT Highlights critical safety concerns for LLM deployment in legal and compliance sectors, emphasizing the need for robust verification mechanisms.
RANK_REASON The cluster contains an academic paper detailing a new benchmark and evaluation of LLM capabilities.
- Arabic
- arXiv
- English
- Gemini 2.5 Flash
- General Data Protection Regulation
- GPT-OSS-120B
- Nemotron-3-Super-120B
- Saudi Personal Data Protection Law
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
