New theory links language models to state probability recovery

This paper introduces a theoretical framework for understanding how observable language probabilities can be used to infer underlying states, particularly in the context of large language models. It proposes a method for identifying a semiparametric inverse from language probabilities to state probabilities, enabling the measurement of states without interpreting the language probabilities as internal beliefs. The research details conditions for existence, recovery, and updating of these state probabilities, along with theoretical rates and minimax boundaries for stability. Simulations and studies using frozen language models are presented to validate the proposed methods. AI

IMPACT Provides a theoretical foundation for interpreting the outputs of large language models as state measurements, potentially improving their scientific utility.

RANK_REASON Academic paper detailing new theoretical methods for statistical inference. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• arXiv
• large language models

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →