Suvadeep Hajra

New research explores transformer instability in long-context models

Two recent arXiv papers delve into the challenges of training transformers with long contexts. The first paper identifies a fundamental pathology called rank-collapse, where attention scores become uniform as context le…