PulseAugur
EN
LIVE 12:27:38

NVIDIA unveils 4-bit pretraining method, NVFP4, for LLMs

By PulseAugur Editorial · [3 sources] ·

NVIDIA has developed a new 4-bit pretraining methodology called NVFP4, designed to overcome the challenges of reduced dynamic range and increased quantization error in narrower floating-point formats. This method was successfully validated by pretraining a 12-billion-parameter hybrid Mamba-Transformer model on 10 trillion tokens, marking the longest publicly documented training run in 4-bit precision to date. The resulting model achieved performance nearly identical to an FP8 baseline on the MMLU-Pro benchmark, demonstrating the viability of NVFP4 for large-scale model training. AI

IMPACT Enables more efficient training of large language models by reducing precision requirements without significant performance loss.

RANK_REASON The cluster describes a new pretraining methodology and its validation on a large model, presented as a research finding.

Read on MarkTechPost →

AI-generated summary · Google Gemini · from 3 sources. How we write summaries →

NVIDIA unveils 4-bit pretraining method, NVFP4, for LLMs

How we ranked this

Signal score
0 / 100
Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket
Research
The cluster describes a new pretraining methodology and its validation on a large model, presented as a research finding.
Source corroboration
3 independent sources
Multiple independent publishers reporting the same story raises confidence that it's real and newsworthy.
Topics
infra, paper
Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance
High
Clearly on-topic for AI-industry coverage.
Story freshness
105 days old
Aged out of breaking-news scoring windows; ranking reflects the durable signal from the full source set.
Coverage growth since scoring
+1 source(s) since last score
New sources have picked up this story since our last re-score. Score will update on the next scoring pass.

Full methodology in our editorial standards.

COVERAGE [3]

  1. MarkTechPost TIER_1 English(EN) · Asif Razzaq ·

    NVIDIA Introduces a 4-Bit Pretraining Methodology Using NVFP4, Validated on a 12B Hybrid Mamba-Transformer at 10T Token Horizon

    <p>NVIDIA introduces a 4-bit pretraining methodology built around the NVFP4 microscaling format — combining selective BF16 layers, 16×16 Random Hadamard Transforms on Wgrad inputs, 2D weight scaling, and stochastic rounding on gradients — validated on a 12B hybrid Mamba-Transform…

  2. Mastodon — sigmoid.social TIER_1 English(EN) · [email protected] ·

    NVIDIA has introduced a 4-bit pretraining methodology using NVFP4, validated on a 12 billion parameter hybrid Mamba-Transformer trained on 10 trillion tokens -

    NVIDIA has introduced a 4-bit pretraining methodology using NVFP4, validated on a 12 billion parameter hybrid Mamba-Transformer trained on 10 trillion tokens - the longest publicly documented 4-bit pretraining run. Accuracy closely matches the FP8 baseline at 62.58% versus 62.62%…

    LINKS marktechpost.com/…/nvidia-introduces-a-4-…
  3. Mastodon — fosstodon.org TIER_1 Polski(PL) · [email protected] ·

    NVIDIA has proven that training models on 10 trillion tokens in 4-bit NVFP4 precision does not cause a drop in quality. This is the foundation for the Blackwell architecture

    NVIDIA udowodniła, że trenowanie modeli na 10 bilionach tokenów w 4-bitowej precyzji NVFP4 nie powoduje spadku jakości. To fundament pod architekturę Blackwell i szansa na radykalne obniżenie kosztów szkolenia AI. # si # ai # sztucznainteligencja # wiadomości # informacje # techn…

    LINKS aisight.pl/…/nvidia-4-bity-trenowanie-gig…

RELATED ENTITIES

RELATED TOPICS