Brief

Two separate teams at Nexus Labs and Prophesee have adopted Bifrost, an open-source gateway, to manage their interactions with multiple large language models. Prophesee used Bifrost to caption 1.2 million robotics frames, achieving a 22% cost saving by intelligently routing requests across GPT-4o, Claude 3.7 Sonnet, and Gemini 2.5 Pro. Nexus Labs implemented Bifrost to improve the quality of their agent training data, finding that nearly half of their production traces were unusable due to inconsistent model behavior and hidden provider failures. By using Bifrost's advanced fallback and logging features, they were able to reduce corrupted traces from 17% to under 3%, enabling more reliable fine-tuning. AI

A new approach to evaluating Large Language Models (LLMs) has been proposed to address the issue of static evaluation harnesses failing to detect model regressions. This method involves refreshing evaluation datasets weekly with real production traces, stratified by intent cluster to ensure representative sampling. Additionally, a permanent adversarial set, curated from actual customer support tickets indicating model failures, is weighted heavily in the evaluation process to prioritize real-world performance. AI