A Reddit user explored whether large language models like Gemma and Qwen can detect their own hallucinations by analyzing token probabilities (logprobs). The experiment suggested that when a model is uncertain, probabilities are distributed across multiple tokens, whereas a confident but incorrect recall might concentrate on a single wrong belief. The user developed a custom WebUI tool to test this, finding that both Gemma and Qwen performed poorly in utilizing their logprobs to identify uncertainty or hallucinations. AI
IMPACT Suggests current methods for LLMs to self-detect hallucinations are limited, impacting reliability.
RANK_REASON User-led exploration of LLM capabilities, not a formal research paper or product release.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
