HRM-Text model drastically cuts LLM pretraining costs

Researchers have developed HRM-Text, a novel Hierarchical Recurrent Model that significantly reduces the computational resources and training data required for pretraining large language models. By decoupling computation into strategic and execution layers and training exclusively on instruction-response pairs, a 1B-parameter model achieved competitive performance on several benchmarks with a fraction of the tokens and compute used by standard models. This approach makes foundational LLM research more accessible by lowering the barrier to entry for pretraining from scratch. AI

影响 Enables more researchers to train foundational models from scratch, potentially accelerating innovation.

排序理由 The cluster contains an academic paper detailing a new model architecture and its performance on benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

• DROP
• GSM8K
• Hierarchical Recurrent Model
• HRM-Text
• MMLU

• 论文
• 模型发布
• 基础设施

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →