Researchers have developed Kierston, a city-scale simulation engine, and KierstonBench, a benchmark designed to evaluate the capabilities of large language models in criminal investigation and surveillance tasks. The simulation models the activities and records of approximately 125,000 residents in Allentown, Pennsylvania, across various data sources. An evaluation of sixteen language models showed that while most struggled with record investigation and often fabricated evidence, a few advanced models could perform most tasks and appropriately abstain when evidence was insufficient, though they still fell prey to deceptions and could not replace human investigators. AI
IMPACT Establishes a new evaluation framework for LLMs in sensitive domains like criminal investigation and surveillance, highlighting current limitations.
RANK_REASON The cluster describes a new benchmark and simulation engine for evaluating language models on specific research tasks.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
