Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
A new paper explores the critical role of graph tokenization in applying Transformers to graph learning tasks. Researchers demonstrate that the method used to convert graph structures into tokens significantly impacts a Transformer's expressivity and the depth required for computations. The study proves that certain tokenizations, like random-walk, are inherently lossy, while others, like spectral tokenization, may be ill-conditioned for specific tasks. The findings suggest that combining complementary tokenization strategies can enhance a Transformer's ability to leverage diverse structural signals for improved performance. AI
IMPACT Highlights how graph tokenization methods fundamentally affect Transformer performance in graph learning tasks.