Looped Transformers
PulseAugur coverage of Looped Transformers — every cluster mentioning Looped Transformers across labs, papers, and developer communities, ranked by signal.
2 day(s) with sentiment data
-
SMELT architecture improves MoE Transformer training efficiency
Researchers have developed SMELT, a new architecture for Mixture-of-Experts (MoE) Transformers that improves training efficiency and downstream performance. By looping the middle layers of the transformer twice while ca…
-
Dynamical phase selection controls compute scaling in looped transformers
Researchers have identified that the computational cost of looped transformers during inference is determined by their dynamical phase, which is influenced by initialization. Networks with identical architectures and ob…
-
New SCSE method improves Looped Transformers for text tasks
Researchers have introduced Source-Centered State Evolution (SCSE), a novel method designed to enhance Looped Transformers. SCSE addresses the challenge of maintaining consistent hidden states across varying recurrent d…
-
Researchers explore adaptive depth and cyclic folding for Transformer optimization
Two new research papers explore novel approaches to optimizing Transformer models by dynamically adjusting their depth. The first paper, "Adaptive Depth in Looped Transformers," investigates learned halting gates and tr…
-
DeepLoop method enhances Looped Transformer depth scaling
Researchers have introduced DeepLoop, a novel method for scaling the depth of Looped Transformers. This technique addresses the challenge of residual scaling in recurrent architectures by formalizing the tied-depth effe…
-
LayerNorm acts as implicit gain control in looped transformers, research finds
A new research paper proposes that Layer Normalization in pre-LayerNorm looped transformers functions as an implicit gain controller. This mechanism helps stabilize the recurrence by coupling the block's local Lipschitz…
-
DiscoLoop architecture enhances multi-hop reasoning in LLMs
Researchers have developed DiscoLoop, a novel looping architecture designed to enhance multi-hop reasoning in large language models. Standard Transformers struggle with retaining information across multiple reasoning st…
-
Looped Transformers Stabilized with Learned Stochastic Stopping
Researchers have developed a method to stabilize extrapolation in Looped Transformers, a type of neural network architecture designed for variable-length algorithmic tasks. While these models can generalize well to long…
-
New research explores latent reasoning for LLMs, offering efficiency and interpretability gains
Two new research papers explore alternative methods for improving reasoning in large language models. One paper introduces LoTUS (Looped Transformers with parallel supervision on latents), a method using recurrent-depth…
-
New research explores merging large transformers and improving looped model stability
Two new research papers explore novel techniques for enhancing the capabilities and stability of large transformer models. The first paper introduces a scalable framework for linear mode connectivity (LMC) that allows f…
-
New LLM techniques enhance reasoning via iterative refinement and optimized looping · 5 sources tracked
Researchers have developed new methods to improve the reasoning capabilities of large language models (LLMs) through test-time scaling. The REVES framework uses a two-stage iterative process to augment training data and…
-
New methods boost LLM reasoning efficiency with compressed CoT
Researchers have developed new methods to improve the efficiency of chain-of-thought (CoT) reasoning in large language models. HybridThinker introduces a training scheme that balances retaining detailed thought steps wi…
-
Looped Transformers with Layer Norm Provably Learn Power Method
Researchers have theoretically demonstrated how looped transformers with layer normalization can learn the power method for principal component prediction. The study proves that such models, when trained with gradient d…
-
New dual-path architecture scales LLM compute and capacity
Researchers have introduced a novel dual-path architecture for large language models designed to efficiently scale both compute and capacity. This architecture utilizes a deep sublayer applied multiple times with shared…
-
Looped Transformers: A New Architecture for Enhanced Language Models
This article introduces the concept of looped transformers, a novel architecture for language models that aims to improve contextual understanding and dynamic representation. It explains how traditional transformer mode…