PulseAugur
EN
LIVE 08:20:38

New research reveals data shuffle order as a major source of fine-tuning noise

By PulseAugur Editorial · [2 sources] ·

A new paper explores how the order in which data is shuffled during the fine-tuning of machine learning models can introduce significant noise. This noise, stemming from the memory of optimizers like AdamW and SGD, can even flip the results of A/B comparisons. The research proposes a method to quantify this noise without fitting parameters, offering insights into order-variance and providing criteria for fine-tuning comparisons. AI

IMPACT Highlights a previously underestimated factor in model training that could impact reproducibility and performance comparisons.

RANK_REASON The cluster contains a single academic paper detailing a new research finding. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

New research reveals data shuffle order as a major source of fine-tuning noise

COVERAGE [2]

  1. arXiv stat.ML TIER_1 English(EN) · John Sweeney ·

    Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

    arXiv:2606.29554v1 Announce Type: cross Abstract: Shuffle order can be a larger source of fine-tuning noise than a memoryless analysis predicts: fixed-clock optimizer memory makes local equal-multiset contrasts first order in the learning rate rather than second order, and the re…

  2. arXiv stat.ML TIER_1 English(EN) · John Sweeney ·

    Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

    Shuffle order can be a larger source of fine-tuning noise than a memoryless analysis predicts: fixed-clock optimizer memory makes local equal-multiset contrasts first order in the learning rate rather than second order, and the resulting order channel can be large enough for a si…

RELATED ENTITIES

RELATED TOPICS