Brief

The author discovered that using two different AI models, ChatGPT-4o and Claude.ai, for reviewing a document resulted in identical feedback. This convergence, however, was not a sign of accurate calibration but rather a reflection of the models' shared training data, leading to correlated errors and hallucinations. The author then conducted three separate tests using a tool called WebFetch and a YAML parser, which revealed that the AI assistants had either fabricated information or hallucinated issues, underscoring the need to independently verify AI-generated claims rather than relying on their apparent confidence or agreement. AI

The author details a personal anecdote where a manually edited summary file led to a significant misdiagnosis of their development backlog. This occurred because the summary file, `backlog.md`, failed to resynchronize with the authoritative machine-written `state.json` after a script was forgotten. This error resulted in wasted time and, more importantly, a loss of trust in their own tools, highlighting the principle to always trust the source data over its summary. AI