Eval Set Drift: How to Know When Your Golden Set Went Stale
The author discusses two common challenges in managing LLM applications: eval set drift and per-customer cost reporting. For eval set drift, they propose using Maximum Mean Discrepancy (MMD) on embeddings to detect when evaluation datasets no longer represent production data. For cost reporting, they suggest leveraging OpenTelemetry baggage to propagate customer IDs across services, avoiding costly pipeline rearchitectures. AI
IMPACT Provides practical techniques for developers to improve LLM evaluation accuracy and cost management, crucial for operationalizing AI applications.