Slot Machines: How LLMs Keep Track of Multiple Entities
Researchers have identified a novel mechanism within language models, termed "slot machines," that enables them to manage multiple entities and their associated attributes simultaneously. This multi-slot probing approach reveals that individual tokens can encode information about both the current and preceding entities, with distinct functional roles for each slot. While the "current-entity" slot is crucial for direct factual retrieval, the "prior-entity" slot aids in relational inferences and conflict detection, though its utility for explicit retrieval is limited in open-weight models. AI
IMPACT Reveals a potential limitation in how LLMs track multiple entities, impacting agentic behavior and complex reasoning.