PulseAugur
EN
LIVE 19:38:12

LLMs struggle to generate multi-GPU kernels, researchers find

By PulseAugur Editorial · [5 sources] ·

Researchers at Together have found that while large language models can efficiently generate single-GPU kernels, they struggle significantly with multi-GPU kernel generation. These models perform poorly when asked to create kernels optimized for multiple GPUs, often failing to compile or producing incorrect results. This limitation stems from the difference in bottlenecks between single-GPU (compute/memory bandwidth) and multi-GPU (interconnect) operations, which current LLMs do not effectively handle. AI

IMPACT Highlights a current limitation in LLM capabilities for complex parallel programming tasks, potentially impacting AI infrastructure development.

RANK_REASON Research findings on LLM capabilities in generating multi-GPU kernels.

Read on X — Together (inference / OSS) →

AI-generated summary · Google Gemini · from 5 sources. How we write summaries →

LLMs struggle to generate multi-GPU kernels, researchers find

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
Research findings on LLM capabilities in generating multi-GPU kernels.
Source corroboration
5 independent sources
Strong cross-source corroboration — multiple independent publishers covered this within the clustering window.
Topics
model release, infra
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
106 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

Full methodology in our editorial standards.

COVERAGE [5]

  1. X — Together (inference / OSS) TIER_1 Dansk(DA) · togethercompute ·

    LLMs are getting better at writing GPU kernels. Multi-GPU kernels are the harder test.

    LLMs are getting better at writing GPU kernels. Multi-GPU kernels are the harder test. At @aiDotEngineer World's Fair, @simran_s_arora will share ParallelKernelBench, an open-source benchmark built from real CUDA communication problems where performance depends on moving data ht…

  2. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute ·

    An agentic loop (compile, test, profile, revise) helps. Gemini 3 Pro went from 24 to 35/87 correct, then plateaued after ~20 steps.

    An agentic loop (compile, test, profile, revise) helps. Gemini 3 Pro went from 24 to 35/87 correct, then plateaued after ~20 steps. Feedback fixes syntax, not rank coordination, collective ordering, or transfer-mechanism choice. TMA and NVLS stay almost unused. https://t.co/VKER…

  3. X — Together (inference / OSS) TIER_1 Dansk(DA) · togethercompute ·

    Frontier models struggle.

    Frontier models struggle. → Best zero-shot: 28/87 correct, 22 beat the PyTorch + NCCL baseline → With 3 attempts: 36/87 correct, but fast1@3 tops out at 31% Weak models fail to compile. Strong reasoners compile cleanly and return wrong answers. https://t.co/1fgoyZRmuH

  4. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute ·

    But why are multi-GPU kernels so different? Single-GPU kernels are bottlenecked by compute and memory bandwidth. Multi-GPU kernels by the interconnect.

    But why are multi-GPU kernels so different? Single-GPU kernels are bottlenecked by compute and memory bandwidth. Multi-GPU kernels by the interconnect. Each PKB task hands the model a PyTorch + NCCL reference and asks it to communicate directly across GPUs via symmetric memory. …

  5. X — Together (inference / OSS) TIER_1 English(EN) · togethercompute ·

    LLMs write fast single-GPU kernels. Ask for a multi-GPU one and they fall apart.

    LLMs write fast single-GPU kernels. Ask for a multi-GPU one and they fall apart. ParallelKernelBench measures how they fail by benchmarking against 87 problems pulled from real codebases including Megatron-LM, DeepSpeed, DeepEP, TensorRT-LLM, NeMo-RL. New research from Willy ht…

RELATED ENTITIES

RELATED TOPICS