PulseAugur
EN
LIVE 17:29:12

Buoyant open-sources Kinference for optimized LLM inference on Kubernetes

By PulseAugur Editorial · [2 sources] ·

Buoyant has released Kinference, an open-source control plane designed to manage large language model inference on Kubernetes clusters using vLLM. This new tool introduces policy-driven routing, token rate limiting, and KV-cache-aware load balancing to optimize model serving. Benchmarks indicate significant improvements in time-to-first-token and goodput, particularly for workloads with shared prefixes, though performance gains are less pronounced for prompts without shared cache data. AI

IMPACT Optimizes LLM inference serving on Kubernetes, potentially improving efficiency and cost for organizations deploying multiple models.

RANK_REASON This is a new open-source tool release for managing LLM inference, not a frontier model release or significant industry shift.

Read on Mastodon — sigmoid.social →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

Buoyant open-sources Kinference for optimized LLM inference on Kubernetes

How we ranked this

Signal score
3 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
This is a new open-source tool release for managing LLM inference, not a frontier model release or significant industry shift.
Source corroboration
2 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
product, infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Same-day
Cluster formed today. Ranking reflects the current source set at time of score.

Full methodology in our editorial standards.

COVERAGE [2]

  1. MarkTechPost TIER_1 English(EN) · Michal Sutter ·

    Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

    <p>Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes. It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark.</p> <p>The post <a href="https://www.m…

  2. Mastodon — sigmoid.social TIER_1 English(EN) · [email protected] ·

    Buoyant, the team behind Linkerd, has open-sourced Kinference, a control plane for running vLLM on Kubernetes. It adds declarative routing policies, token rate

    Buoyant, the team behind Linkerd, has open-sourced Kinference, a control plane for running vLLM on Kubernetes. It adds declarative routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics. In Buoyant's own benchmark, median time to first token fel…

    LINKS marktechpost.com/…/buoyant-kinference-ope…

RELATED ENTITIES

RELATED TOPICS