Your RAG dashboard can hide a failing retriever: detecting silent regression
A new tool called eval-sanity v0.2 has been released to detect silent regressions in Retrieval-Augmented Generation (RAG) systems. These regressions occur when the retriever component degrades, causing it to miss relevant documents, but the generator continues to produce fluent answers from the partial context, masking the issue on standard dashboards. The tool uses statistical analysis of evaluation runs to differentiate significant drops in retrieval from normal metric fluctuations, preventing false alarms and alerting users to hidden performance degradation. AI
IMPACT Helps AI operators maintain RAG system performance by identifying subtle degradation issues.