A new research paper proposes that Large Language Models (LLMs) can be considered Bayesian predictors, even if their internal mechanisms don't perfectly align with traditional Bayesian expectations. The study suggests that while exact posterior predictives in Bayesian accounts are invariant to task-preserving orderings, transformers can alter next-token probabilities based on serialization order. However, the paper argues that this deviation doesn't invalidate their Bayesian competitiveness, as the excess prequential code length is directly related to predictive KL divergence. Experiments on Qwen2.5 models show that their predictive distributions closely resemble Bayesian posterior predictives, particularly at smaller support sizes, and that positional encoding is a key factor in order sensitivity. AI
IMPACT This research offers a theoretical framework for understanding LLM behavior, potentially guiding future model development towards more robust and predictable performance.
RANK_REASON The cluster contains an academic paper discussing theoretical aspects of LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
