Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy
Researchers have developed a new metric called conditional scale entropy (CSE) to analyze how decoder-only language models process metaphors. CSE measures the breadth of computational engagement across different frequency scales within a transformer's layers. Studies using CSE revealed that metaphorical tokens consistently activate a wider range of computational scales compared to literal tokens in models ranging from 124 million to 20 billion parameters, including architectures like GPT-2, LLaMA-2, and GPT-oss. AI
IMPACT Introduces a novel metric for understanding metaphorical processing in LLMs, potentially aiding in the development of more nuanced language understanding capabilities.