Formalizing and Mitigating Structural Distortion in LLM Attention for Zero-Shot Graph Reasoning
Researchers have identified a key mechanism, termed structural distortion, that hinders Large Language Models (LLMs) from effectively reasoning over text-attributed graphs. This distortion arises from the linearization of graph structures into sequences, which, when combined with rotary positional embeddings, causes attention decay between adjacent nodes that are distant in the sequence. To address this, a new method called Graph-aligned Language Attention (GaLA) has been proposed. GaLA is an inference-time modification that biases LLM attention towards graph-adjacent nodes without significant overhead, improving performance on graph reasoning benchmarks. AI