feed-forward networks
PulseAugur coverage of feed-forward networks — every cluster mentioning feed-forward networks across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New dataset and deep learning models enable contactless weight estimation
Researchers have developed a new computer vision approach for contactless weight estimation of falling particles, addressing limitations of traditional scales. They introduced Doppio, a dataset featuring videos of groun…
-
Long-context training may harm LLM knowledge, study finds
A new research paper introduces the "Information Abundance Paradox," challenging the assumption that longer context windows in large language models always improve performance. The study suggests that excessive relevant…
-
New method optimizes Transformer FFN width using geometric analysis
Researchers have developed a novel method for optimizing the width of feed-forward networks (FFNs) within Transformer models, moving away from the standard constant width. By analyzing the geometric changes in token rep…
-
Apple unveils MemoryLLM for interpretable Transformer FFNs
Apple's Machine Learning Research team has introduced MemoryLLM, a novel approach to enhance the interpretability of feed-forward networks (FFNs) within Transformer models. By decoupling FFNs from self-attention mechani…
-
New research reveals loss-critical channels in LLM feed-forward layers
Researchers have identified a specific organizational structure within the feed-forward layers of Large Language Models (LLMs), termed "supernodes" and "halos." These supernodes represent a small percentage of channels …