PulseAugur
EN
LIVE 13:15:17

New mechanistic estimation method outperforms sampling for wide random MLPs

By PulseAugur Editorial · [4 sources] ·

Researchers have developed a new method for estimating the expected output of wide, randomly initialized multilayer perceptrons (MLPs) without needing to run samples through the model. This "mechanistic estimation" approach leverages tools like cumulants and Hermite expansions to provide more accurate results than traditional Monte Carlo sampling, especially for wide networks. The technique is also more efficient, requiring fewer floating-point operations (FLOPs) and showing particular promise for estimating rare events and for use in model training, potentially reducing catastrophic tail risks. AI

IMPACT Offers a more efficient and potentially safer method for training models, especially for mitigating rare but high-impact risks.

RANK_REASON This is a research paper detailing a new theoretical and empirical method for estimating MLP outputs.

Read on arXiv stat.ML →

AI-generated summary · Google Gemini · from 4 sources. How we write summaries →

New mechanistic estimation method outperforms sampling for wide random MLPs

COVERAGE [4]

  1. Alignment Forum TIER_1 English(EN) · Jacob_Hilton ·

    Mechanistic estimation for wide random MLPs

    <p><em>This post covers joint work with Wilson Wu, George Robinson, Mike Winer, Victor Lecomte and Paul Christiano. Thanks to Geoffrey Irving and Jess Riedel for comments on the post.</em></p> <p>In ARC's latest paper, we study the following problem: given a randomly initialized …

  2. arXiv cs.LG TIER_1 English(EN) · Wilson Wu, Victor Lecomte, Michael Winer, George Robinson, Jacob Hilton, Paul Christiano ·

    Estimating the expected output of wide random MLPs more efficiently than sampling

    arXiv:2605.05179v1 Announce Type: new Abstract: By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initializati…

  3. LessWrong (AI tag) TIER_1 English(EN) · Jacob_Hilton ·

    Mechanistic estimation for wide random MLPs

    <p><em>This post covers joint work with Wilson Wu, George Robinson, Mike Winer, Victor Lecomte and Paul Christiano. Thanks to Geoffrey Irving and Jess Riedel for comments on the post.</em></p> <p>In ARC's latest paper, we study the following problem: given a randomly initialized …

  4. arXiv stat.ML TIER_1 English(EN) · Paul Christiano ·

    Estimating the expected output of wide random MLPs more efficiently than sampling

    By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, we show how to estimate its expected output …

RELATED ENTITIES

RELATED TOPICS