Order-agnostic language models show order-dependent likelihoods

Researchers have identified that order-agnostic language models (OALMs) do not perfectly factorize joint distributions, meaning the order in which tokens are revealed can impact the generated likelihood by up to 0.49 nats/token. While confidence-first decoding is order-agnostic, its token reveal order closely resembles left-to-right generation. The study also proposes a new diagnostic tool based on the variance of confidence traces, showing that uniform confidence spreading maximizes target recoverability and that lower variance correlates with higher downstream correctness. AI

IMPACT Reveals fundamental limitations in order-agnostic language models, potentially guiding future research in decoding strategies and model evaluation.

RANK_REASON Academic paper detailing novel findings about language model behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• LLaDA-2.1

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →