Dyck language

Transformer models' hierarchical representations are decodable but not always causally used

Researchers have investigated how transformer models represent hierarchical structures, specifically in tasks involving bracket sequences. They found that while information about hierarchy, such as depth and position, i…