Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading
Researchers have identified that order-agnostic language models (OALMs) do not perfectly factorize joint distributions, meaning the order in which tokens are revealed can impact the generated likelihood by up to 0.49 nats/token. While confidence-first decoding is order-agnostic, its token reveal order closely resembles left-to-right generation. The study also proposes a new diagnostic tool based on the variance of confidence traces, showing that uniform confidence spreading maximizes target recoverability and that lower variance correlates with higher downstream correctness. AI
IMPACT Reveals fundamental limitations in order-agnostic language models, potentially guiding future research in decoding strategies and model evaluation.