Building independent LLM drift detection - sharing the methodology, looking for feedback on the approach
A developer is proposing a new service to detect silent degradation in large language model (LLM) performance, beyond standard API health checks. The proposed tool would run external canary tests to monitor aspects like JSON adherence, instruction following, and refusal behavior, comparing current model outputs against historical baselines and peer models. The developer is seeking feedback on the technical soundness, valuable alert types, and potential pricing for such a service, particularly for agentic systems where subtle performance shifts can lead to significant operational failures. AI
IMPACT Could improve reliability and trust in LLM deployments, especially for agentic systems.