Nexus Labs
PulseAugur coverage of Nexus Labs — every cluster mentioning Nexus Labs across labs, papers, and developer communities, ranked by signal.
- 2026-05-29 product_launch Nexus Labs implemented a multi-LoRA serving strategy using vLLM to reduce infrastructure costs for enterprise AI deployments. source
6 day(s) with sentiment data
Nexus Labs' new evaluation strategy may become a de facto standard for evaluating LLM agents.
Nexus Labs discovered that aggregate evaluation scores can mask critical performance regressions in specific customer segments. Their new strategy, which stratifies results by customer segment and gates deployments on the worst-performing slice, addresses this issue. This more robust evaluation methodology could be adopted by other organizations seeking to ensure reliable performance across diverse user bases.
Nexus Labs is actively optimizing vLLM for diverse LLM workloads, encountering both successes and challenges.
Recent evidence shows Nexus Labs experimenting with vLLM's continuous batching and prefix caching. While prefix caching significantly reduced latency for consistent system prompts, continuous batching led to p99 latency spikes. This suggests Nexus Labs is pushing the boundaries of vLLM but requires careful tuning and workload-specific configurations to achieve optimal performance across all use cases.
Nexus Labs will announce a new product or service leveraging their multi-LoRA serving capability within 90 days.
Nexus Labs has demonstrated a cost-effective method for serving 40 LoRA adapters on a single Llama 3.1 model, significantly reducing infrastructure costs. This capability is particularly valuable for enterprise deployments with diverse customer needs. It is plausible they will productize this efficiency gain into a new offering or a feature of an existing service to attract customers seeking cost-optimized LLM solutions.
-
LLM Evals Get Granular with Bifrost Request Tagging
A new method for evaluating Large Language Models (LLMs) has been introduced, utilizing request tagging with Bifrost dimension headers. This approach attaches metadata like checkpoint and run IDs to each LLM API call, e…
-
LLM judges show 18% position bias; dual-pass scoring cuts error rate
A study by Nexus Labs revealed that Large Language Models (LLMs) used as judges exhibit significant position bias, favoring the first answer presented in 18% of comparisons. This bias was observed across models like GPT…
-
LLM evaluation metrics need confidence intervals to distinguish signal from noise
Evaluating Large Language Models (LLMs) requires understanding the uncertainty inherent in performance metrics. A single score, such as 84.2% accuracy, can be misleading because it doesn't account for sampling error. By…
-
Gateway simplifies LLM benchmarking across multiple providers
Nexus Labs developed a gateway called Bifrost to streamline benchmarking of multiple Large Language Models (LLMs). By routing requests through a single OpenAI-compatible endpoint, Bifrost simplifies the integration proc…
-
LLM eval reproducibility issues traced to batching and silent routing
Reproducibility issues in LLM evaluations have been identified, stemming not from sampling parameters like temperature, but from underlying inference engine behavior and provider routing. Specifically, floating-point va…
-
Quantization causes 7-point task accuracy drop, bypassing perplexity
A company called Nexus Labs discovered that quantizing a fine-tuned 14B agent model to INT4 using GPTQ resulted in a significant 7-point drop in multi-step task completion accuracy, despite perplexity metrics showing on…
-
New methods boost LLM inference speed via speculative decoding · 7 sources tracked
Researchers are developing advanced speculative decoding techniques to accelerate large language model (LLM) inference. JetFlow, a new framework, improves speed by combining drafting efficiency with causal conditioning,…
-
ML data contamination inflates Qwen3-8B model performance by 9 points
A machine learning team at Nexus Labs discovered that a significant performance increase in their fine-tuned Qwen3-8B model was due to data contamination. The model achieved an 80.4% accuracy on a ticket-routing task, a…
-
Nexus Labs team learns small eval gains are often statistical noise
A machine learning team at Nexus Labs discovered that a recent model promotion was based on a statistically insignificant performance gain. Their internal evaluation suite, which uses exact-match checks, showed a 2.1-po…
-
Dev team hit by silent LLM provider model drift
A software engineering team experienced a significant drop in their automated regression evaluation scores due to silent model updates from a third-party provider. The team discovered that the model they were using was …
-
Nexus Labs agent eval hides 14-point regression in key customer segment
A fine-tuning team at Nexus Labs discovered that their aggregate evaluation scores for an AI agent were misleading, masking a significant performance drop for a specific customer segment. Despite an overall pass rate th…
-
Nexus Labs cuts costs by serving 40 LoRA adapters on one Llama 3.1 model
Nexus Labs has developed a cost-effective method for serving multiple LoRA adapters on a single base model, significantly reducing infrastructure expenses. By utilizing vLLM's multi-LoRA serving capability, they consoli…
-
Fine-tuned Llama 3.1 8B outperforms GPT-4o-mini on invoice extraction
Nexus Labs conducted a shadow test comparing a fine-tuned Llama 3.1 8B model against OpenAI's gpt-4o-mini for invoice line-item extraction. The fine-tuned model demonstrated superior accuracy by 1.8 points and reduced p…
-
vLLM continuous batching causes p99 latency spikes for Llama 3.3
A developer at Nexus Labs encountered significant latency issues after enabling continuous batching in vLLM for their Llama 3.3 70B model. While throughput initially improved, p99 latency increased eightfold, impacting …
-
Nexus Labs replaces 60% of LLM middleware with Bifrost virtual keys
Nexus Labs significantly reduced its custom LLM middleware by replacing over 60% of its 11,247 lines of Python code with Bifrost's virtual key system. This change streamlined per-tenant cost attribution, rate limiting, …
-
LLM judge variance nearly derailed Nexus Labs' agent training
Nexus Labs encountered a significant issue during their DPO training for booking agents, where the LLM used as a preference judge exhibited high self-disagreement (up to 28%), leading to a 4-point drop in production acc…
-
vLLM prefix caching slashes AI agent latency at Nexus Labs
Nexus Labs significantly improved inference latency for their AI agents by implementing vLLM's prefix caching feature. This optimization reduced the time-to-first-token (TTFT) from an average of 410ms to 110ms for tenan…
-
Bifrost gateway improves LLM cost, data quality for robotics and agents
Two separate teams at Nexus Labs and Prophesee have adopted Bifrost, an open-source gateway, to manage their interactions with multiple large language models. Prophesee used Bifrost to caption 1.2 million robotics frame…
-
LLM evaluation harness updated with production data and adversarial testing
A new approach to evaluating Large Language Models (LLMs) has been proposed to address the issue of static evaluation harnesses failing to detect model regressions. This method involves refreshing evaluation datasets we…
-
Measuring AI Gateway Failover: 30 Days of Production Data
Anthropic has released an update on Claude's sycophancy, noting that Opus 4.7 shows a 50% reduction in sycophantic responses compared to Opus 4.6, particularly in relationship guidance conversations. The company also de…