Attention Collapse
PulseAugur coverage of Attention Collapse — every cluster mentioning Attention Collapse across labs, papers, and developer communities, ranked by signal.
-
Split Federated Fine-tuning faces Depth-Performance Dilemma, research finds
A new research paper introduces the "Depth-Performance Dilemma" in Split Federated Fine-tuning (SFF) for Large Language Models (LLMs). This dilemma highlights that while deeper model partitions in SFF can increase syste…
-
New D-SCAN method detects RAG poisoning via attention collapse
Researchers have developed a new method called D-SCAN to detect poisoning attacks in retrieval-augmented generation (RAG) systems. These attacks involve injecting malicious documents into the RAG's context to manipulate…
-
New Bayesian Penalty Reverses Attention Collapse in Language Models
Researchers have developed a new framework called Bayesian Repetition Penalty to address the issue of attention collapse in autoregressive language models. This pathology causes models to get stuck in repetitive loops. …