Jiaqi Leng

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models

Researchers have identified three key design principles crucial for enhancing length generalization in hierarchical sparse attention models. These principles include using an expressive Chunk Encoder with a CLS token fo…