The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
A systematic review of 337 articles indicates that Transformer-based language models (TLMs) possess a significant amount of syntactic knowledge. While these models perform well on formal syntactic tasks, their performance is weaker at the syntax-semantics interface and for less digitally supported languages. Despite evidence of syntactic knowledge, current research methods are too varied and observational to fully understand the underlying computational mechanisms, with a heavy concentration on English and BERT-like models. AI
IMPACT Confirms that current LLMs possess substantial syntactic knowledge, though understanding of the underlying mechanisms remains limited.