PulseAugur
EN
LIVE 20:28:20

Cohere unveils new LLM serving system with "megakernel" architecture

By PulseAugur Editorial · [4 sources] ·

Cohere has introduced a new LLM serving system built around a "megakernel" architecture, which fuses the entire LLM decode step into a single kernel launch. This innovation aims to maximize GPU utilization and improve performance. The system, named North Mini Code, reportedly achieves up to 1.58x faster performance compared to vLLM in certain benchmarks and is fully open-source. AI

IMPACT This development could lead to more efficient LLM serving infrastructure, potentially lowering costs and improving inference speeds.

RANK_REASON This is a product/infrastructure announcement from an AI company, not a frontier model release or core research.

Read on X — Cohere →

AI-generated summary · Google Gemini · from 4 sources. How we write summaries →

Cohere unveils new LLM serving system with "megakernel" architecture

How we ranked this

Signal score
11 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Tool
This is a product/infrastructure announcement from an AI company, not a frontier model release or core research.
Source corroboration
4 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
infra, product
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
Breaking (< 6h)
Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

Full methodology in our editorial standards.

COVERAGE [4]

  1. X — Cohere TIER_1 English(EN) · cohere ·

    Don’t know what a megakernel is? Don’t worry. Find out what it is - and how we built the system - on our blog: https://t.co/HF93hrlynu

    Don’t know what a megakernel is? Don’t worry. Find out what it is - and how we built the system - on our blog: https://t.co/HF93hrlynu

  2. X — Cohere TIER_1 English(EN) · cohere ·

    Some results: With North Mini Code (BF16 on 1×H100), we achieved 1.58x vs vLLM at BS=1 and 1.25x–1.41x end-to-end serving at BS=8 https://t.co/mVE8weG06A

    Some results: With North Mini Code (BF16 on 1×H100), we achieved 1.58x vs vLLM at BS=1 and 1.25x–1.41x end-to-end serving at BS=8 https://t.co/mVE8weG06A

  3. X — Cohere TIER_1 English(EN) · cohere ·

    A megakernel fuses the entire LLM decode step into a single kernel launch. We build on this by maximizing GPU utilization through kernel fusion while supporting

    A megakernel fuses the entire LLM decode step into a single kernel launch. We build on this by maximizing GPU utilization through kernel fusion while supporting everything a real server needs. Check out how we got there on GitHub: https://t.co/f0P7dkv89S

  4. X — Cohere TIER_1 English(EN) · cohere ·

    Introducing the next evolution in LLM text generation: the first fully-fledged serving system built around a decode megakernel.

    Introducing the next evolution in LLM text generation: the first fully-fledged serving system built around a decode megakernel. Delivering up to 1.58x faster performance than vLLM. Built for North Mini Code, completely open-source.

RELATED ENTITIES

RELATED TOPICS