LLMs Form Implicit Discrete State Representations for Internal Calculation

Researchers have identified that Large Language Models (LLMs) can perform extended calculations internally without explicit chain-of-thought reasoning. This suggests that models may form Implicit Discrete State Representations (IDSRs) within their hidden states to conduct symbolic calculations. The study found that while these representations are used for internal computation, they are not perfectly lossless in current open-source models, leading to errors in final outputs. This research provides an initial exploration into the symbolic calculation abilities and underlying mechanisms of LLMs. AI

IMPACT Suggests potential for improved internal reasoning and calculation capabilities in future LLMs.

RANK_REASON Research paper detailing emergent capabilities in LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Hugging Face
• IDSRs
• Implicit Discrete State Representations
• Junhao Chen
• LLMs

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →