New framework aims for verifiable, bias-aware LLM user simulators

A new tutorial proposes a framework for creating verifiable user simulators based on large language models (LLMs). This framework aims to address the opacity of current LLM-based simulators, making it easier to understand why a simulated user makes certain choices and to detect potential biases related to demographic characteristics. The proposed system includes components for structured personas, task-aware contracts, auditable traces, and persona-aligned verification, with a refinement loop for continuous improvement. Hands-on labs will allow participants to evaluate simulator behavior for fidelity, credibility, and demographic bias in recommendation and search tasks. AI

IMPACT This framework could improve the reliability and fairness of AI evaluations by making user simulators more transparent and auditable.

RANK_REASON The cluster contains an academic paper detailing a new framework for verifiable user simulation in AI systems. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.IR (Information Retrieval) →

• arXiv
• Information Retrieval
• Large-language-model
• LLM

• paper
• safety
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →