Laura Gomezjurado Gonzalez

Transformer grokking delay linked to decoder bottleneck, study finds

A new research paper explores the phenomenon of 'grokking' in transformers, where models abruptly generalize after a long delay during training on algorithmic tasks. The study suggests this delay stems from limited acce…