PulseAugur
EN
LIVE 10:11:13

Fireworks AI details complex RL infrastructure for continuous model updates

By PulseAugur Editorial · [10 sources] ·

Fireworks AI is detailing the engineering challenges and solutions involved in training large language models, particularly focusing on Reinforcement Learning (RL) from human feedback. They highlight that a product's real-world usage is the most effective RL environment, emphasizing the need for infrastructure that can continuously update models based on live user interactions. The company also discusses the complexities of distributed RL, including numerical stability issues and the efficient syncing of massive model weights across global clusters. AI

IMPACT Fireworks AI's insights highlight the significant engineering effort required for advanced model training, particularly in RL, suggesting that efficient infrastructure is key to continuous improvement.

RANK_REASON The cluster consists of a series of X posts from Fireworks AI detailing their engineering approach to model training and RL, rather than a direct product or model release.

Read on X — Fireworks (inference infra) →

AI-generated summary · Google Gemini · from 10 sources. How we write summaries →

COVERAGE [10]

  1. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    10/ The bigger point: your product is the best RL environment you'll ever have.

    10/ The bigger point: your product is the best RL environment you'll ever have. Frontier labs ship models that are good at everything. The opportunity is a model that's great at your thing. Product, users, harness. That's the moat. Check out the ep: https://t.co/j085PLDElj

  2. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    9/ Real-time RL is where it gets fun.

    9/ Real-time RL is where it gets fun. Catch live signals from real users on real generations. Update continuously. Ship a new version every few hours. Only works if the base model is already good enough that people want to use it. Real-time RL is the amplifier that runs on a

  3. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    8/ Models cheat. RL rewards cheating.

    8/ Models cheat. RL rewards cheating. They figure out when they're in a sim versus production, and they learn tricks that score well in fake environments but fail for real. The RL environment has to look like production, or you're training a model that games the eval.

  4. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    7/ There's a quiet numerical problem buried in distributed RL that will wreck a run.

    7/ There's a quiet numerical problem buried in distributed RL that will wreck a run. Floating point addition isn't associative, so inference and training produce slightly different log probs for the same tokens. In an MoE model, a tiny difference can flip which expert activates,

  5. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    6/ Syncing 1TB of weights across four global clusters every 5 to 10 minutes is its own engineering problem.

    6/ Syncing 1TB of weights across four global clusters every 5 to 10 minutes is its own engineering problem. Trick is that RL only updates a subset of weights per step. A lossless delta compression scheme shrinks the transfer about 20x. Weights ship in under a minute. Inference

  6. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    5/ The thing that made the math work was async (pipelined) RL.

    5/ The thing that made the math work was async (pipelined) RL. Naive RL pauses training while rollouts run. Half the GPUs sit idle. Pipelined RL runs trainer and rollout workers at the same time. You eat a little staleness, but utilization goes way up. The bitter lesson wins

  7. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    4/ RL infrastructure is harder to build than pre-training infrastructure. A lot harder.

    4/ RL infrastructure is harder to build than pre-training infrastructure. A lot harder. Pre-training needs a big cluster. RL needs a big cluster, plus a whole inference fleet running rollouts that look like what users actually do. A rollout here is a full 50-turn Cursor agent

  8. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    3/ Two pushes got them there.

    3/ Two pushes got them there. Mid-training on code at near pre-training scale to teach the model to write code. Large-scale RL on top to teach it to write correct code. Both required.

  9. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    2/ The mental model Federico opens with is the one that reframes everything.

    2/ The mental model Federico opens with is the one that reframes everything. A model is a storage drive. Finite bits. You decide what goes in. Cursor cares about software engineering inside Cursor. Spend every bit on that one job, and the model ends up running roughly 10x

  10. X — Fireworks (inference infra) TIER_1 English(EN) · FireworksAI_HQ ·

    1/ Composer 2.5 is having a moment. Worth a look at how the team actually got here.

    1/ Composer 2.5 is having a moment. Worth a look at how the team actually got here. @cursor_ai's Federico Cassano and @FireworksAI_HQ cofounder Dima Dzhulgakov discussed Training Data with @sonyatweetybird. The whole episode is worth your time, but we’ll break it down here.

RELATED ENTITIES

RELATED TOPICS