The Allen Institute for Artificial Intelligence (Ai2) has released Olmo-core 3, an open-source training framework for its Olmo language models. This new version introduces a distributed data parallelism approach, replacing the previous fully sharded data parallelism. This change significantly boosts throughput by ensuring each GPU retains its model partition and only selected token rows are sent to the relevant expert, thereby reducing the "capacity tax" associated with storing all parameters. AI
IMPACT This framework's new parallelism strategy could lead to more efficient training of large mixture-of-experts models, potentially lowering costs and accelerating development.
RANK_REASON Release of an open-source training framework for language models with technical details on performance improvements. [lever_c_demoted from research: ic=1 ai=1.0]
- ai2-olmo-core
- Allen Institute for Artificial Intelligence
- Apache Software License 2.0
- Hugging Face
- NVIDIA B300
- Olmo
- Olmo-3-32B
- OLMo 3 7B
- Olmo-core 3
- Python Package Index
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →