Brief

The author discusses two common challenges in managing LLM applications: eval set drift and per-customer cost reporting. For eval set drift, they propose using Maximum Mean Discrepancy (MMD) on embeddings to detect when evaluation datasets no longer represent production data. For cost reporting, they suggest leveraging OpenTelemetry baggage to propagate customer IDs across services, avoiding costly pipeline rearchitectures. AI

Embeddings are a core concept in AI, transforming text and other data into numerical representations that capture meaning. These numerical vectors allow AI models to understand relationships between words and concepts, enabling functionalities like semantic search and Retrieval-Augmented Generation (RAG). While vector databases like Pinecone, Weaviate, and Chroma are commonly used for storing and querying these embeddings, alternative approaches like BM25 retrieval with tools such as Meilisearch can also be effective for specific use cases, offering simpler operation and lower costs. AI