NeuronFabric architecture enables on-chip transformer training

Researchers have introduced NeuronFabric, a software reference architecture designed for on-chip transformer training using local Adam updates. A C# prototype demonstrates the feasibility of this approach, handling forward pass, backpropagation, and Adam optimization without external frameworks. The architecture aims to reduce memory requirements by storing weights in BF16 while keeping Adam optimizer moments in FP32, a configuration termed BF16W. This method was validated on a 334K-parameter transformer trained on the Shakespeare corpus, showing comparable evaluation loss to an FP32 GPU reference. AI

IMPACT Proposes a novel architecture for efficient on-chip transformer training, potentially reducing hardware memory requirements.

RANK_REASON Research paper introducing a novel software architecture for on-chip transformer training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Adam
• application-specific integrated circuit
• BF16W
• field-programmable gate array
• NeuronFabric
• Shakespeare
• Xilinx ZCU102

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →