When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE
Two new arXiv papers explore the statistical and geometric properties of the softmax function, a core component in many AI models. The first paper, "When Softmax Fails at the Top," introduces WEINCE, a modification to contrastive learning objectives that improves performance on vision benchmarks by addressing statistical misalignments. The second paper, "The Information Geometry of Softmax," delves into how AI systems encode semantic structure in their representation spaces, proposing "dual steering" as a method to control and stabilize concept manipulation in representations that define softmax distributions. AI
IMPACT These papers offer theoretical insights into the fundamental mechanisms of AI models, potentially leading to more robust and controllable representations.