New research probes LLMs' ability to distinguish own output from user input

A new research paper explores the concept of "reality monitoring" in large language models (LLMs), which is the ability to distinguish between self-generated content and user input. The study found that LLM performance in source attribution is highly dependent on how conversational memory is structured. When memory demands are minimal, LLMs show high accuracy for self-generated content, but this accuracy becomes fragile when episodic memory is introduced, leading to a preference for external items. The research highlights two failure modes: some models swap internal and external judgments, while others show improved accuracy but decoupled confidence, issues not detected by current benchmarks. This suggests that for autonomous AI systems, tracking the origin of knowledge is as crucial as evaluating the knowledge itself. AI

IMPACT This research could lead to new evaluation methods for LLMs, improving their reliability in conversational and autonomous roles by ensuring they don't confabulate based on their own outputs.

RANK_REASON Academic paper detailing a new concept and experimental findings in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Hugging Face
• large-language models
• Reality monitoring

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →