Sohan Venkatesh

LLMs fail word counting due to specific MLP block, not representation limits

Researchers have identified a specific mechanism within large language models that causes them to fail at counting repeated words, despite correctly encoding the count in their internal representations. A multi-layer pe…