Suvadeep Hajra

Dense local dependencies cause Transformer training instability in long sequences

A new research paper identifies dense local dependencies as a primary cause of training instability in autoregressive transformer language models when processing long sequences, especially with low-precision arithmetic.…