NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam
Researchers have introduced NeuronFabric, a software reference architecture designed for on-chip transformer training using local Adam updates. A C# prototype demonstrates the feasibility of this approach, handling forward pass, backpropagation, and Adam optimization without external frameworks. The architecture aims to reduce memory requirements by storing weights in BF16 while keeping Adam optimizer moments in FP32, a configuration termed BF16W. This method was validated on a 334K-parameter transformer trained on the Shakespeare corpus, showing comparable evaluation loss to an FP32 GPU reference. AI
IMPACT Proposes a novel architecture for efficient on-chip transformer training, potentially reducing hardware memory requirements.