Preisach Attention: A Hysteretic Model of Sequential Memory
Researchers have introduced the Preisach Attention Layer (PAL), a new sequence modeling architecture inspired by the Preisach hysteresis operator from mathematical physics. PAL replaces standard softmax attention with a binary relay operator, maintaining a stack of local extrema as its state. This novel approach offers Turing completeness in a single layer and computes historical range statistics efficiently, making it suitable for tasks requiring long episodic memory with weak positional dependence. AI
IMPACT Introduces a novel architecture for sequence modeling that could improve efficiency for tasks with long-term memory requirements.