TinyShakespeare
PulseAugur coverage of TinyShakespeare — every cluster mentioning TinyShakespeare across labs, papers, and developer communities, ranked by signal.
2 day(s) with sentiment data
-
New HySPE positional encoding method shows superior extrapolation over RoPE
Researchers have introduced Hyperbolic Symplectic Positional Encoding (HySPE), a novel method for grounding positional attention in non-compact symplectic transformations. Unlike Rotary Position Embedding (RoPE) which u…
-
New FourierQK attention mechanism enhances transformer models
Researchers have developed FourierQK, a novel attention mechanism for generative pre-trained transformers that utilizes bandpass-filtered inner products. Experiments on character-level language modeling with TinyShakesp…
-
FourierQK technique boosts transformer attention with spectral preprocessing · 2 sources tracked
Researchers have developed a novel technique called FourierQK that significantly enhances transformer attention mechanisms by applying spectral preprocessing to query-key projections. This method, tested on character-le…
-
LoRA-Muon: New Optimizer Boosts Deep Learning Fine-Tuning Efficiency
Researchers have introduced LoRA-Muon, an optimization technique designed to improve the efficiency and effectiveness of Low-Rank Adaptation (LoRA) for deep learning models. This new method applies spectral steepest-des…
-
Morlet Wavelet Framework Enhances Transformer Positional Encoding
Researchers have introduced Morlet Positional Encoding (MoPE) as a novel framework for Transformer positional encoding, moving beyond traditional sinusoidal and rotary methods. MoPE utilizes the Morlet wavelet to simult…
-
Energy-Gated Attention enhances Transformer models by prioritizing salient tokens
Researchers have introduced Energy-Gated Attention (EGA), a novel mechanism designed to improve transformer models by focusing on spectrally salient tokens. This approach mimics principles from fluid dynamics, prioritiz…