transformer
PulseAugur coverage of transformer — every cluster mentioning transformer across labs, papers, and developer communities, ranked by signal.
- developed by Google Brain 100%
- developed by Noam Shazeer 100%
- instance of Nemotron 3 Nano Omni 95%
- instance of My Little Pony: Friendship Is Magic 90%
- used by Rope 90%
- uses CNN 90%
- uses Rope 90%
- authored by Attention Is All You Need 90%
- instance of Attention Is All You Need 90%
- used by few-shot learning 90%
- used by electroencephalography 80%
- competes with State space models: Univariate representation of a multivariate model, partial interpolation and periodic convergence 80%
- 2026-05-25 research_milestone A new Transformer-based architecture achieved high accuracy in real-time earthquake magnitude classification. 来源
- 2026-05-19 research_milestone A new paper details the discovery of a geometric mechanism for Bayesian inference within transformer architectures. 来源
- 2026-05-08 research_milestone Researchers published a paper establishing approximation error bounds for Transformers on the Hölder class. 来源
17 天有情绪数据
-
ITGPT model tackles irregular timeseries data with generative pretraining
Researchers have developed ITGPT, a novel attention-based architecture designed to process multimodal and irregularly sampled timeseries data. This model can be trained using both self-supervised learning and generative…
-
Shipping logistics boosted by new retrieval-enhanced Transformer model
Researchers have developed a novel deep learning framework called CCRE to improve multi-step port-of-call sequence prediction in global shipping logistics. This framework utilizes a retrieval-enhanced historical encoder…
-
New theory explains Transformer generalization delay via Bayesian inference
Researchers have proposed a new theory explaining why Transformer models delay generalization after memorizing training data. The theory frames attention mechanisms as implicit Bayesian posteriors over task dependency g…
-
RoPE positional embeddings fail in long-context models, study finds
A new theoretical analysis reveals fundamental limitations in Rotary Positional Embeddings (RoPE) when used in Transformer models designed for long contexts. The research proves that as context length grows, RoPE's abil…
-
New Transformer Model Enhances Cellular Network PRB Forecasting
Researchers have developed PRB-RUPFormer, a novel probabilistic Transformer model designed to forecast residual Physical Resource Blocks (PRBs) in cellular networks. This model uniquely processes multivariate KPI time s…
-
MetaBackdoor attack exploits LLM positional encoding for novel vulnerabilities
Researchers have identified a novel vulnerability in large language models, termed MetaBackdoor, which exploits positional encoding rather than textual content for activation. This attack leverages the model's inherent …
-
SAGE3D模型通过新颖的注意力机制增强3D LiDAR角点检测
研究人员推出SAGE3D，这是一种新颖的基于Transformer的模型，用于检测LiDAR数据的3D点云中的角点。该模型采用分层编码器-解码器架构，并包含两项关键创新：软引导注意力（Soft-Guided Attention），在训练过程中利用地面真实标签来优化注意力；以及激励图神经网络（Excitatory Graph Neural Network），通过正向消息传递来提升高置信度角点预测。这种混合方法旨在提高多尺度角点检测的精度和召回率。
-
Smartwatch frameworks detect psychotic relapse using AI
Researchers have developed two smartwatch-based frameworks for detecting psychotic relapse. The first framework forecasts cardiac dynamics, while the second uses a multi-task approach to fuse sleep, motion, and cardiac …
-
New H3D-MarNet framework enhances CT image quality for radiotherapy
Researchers have developed H3D-MarNet, a novel two-stage framework designed to improve CT image quality for radiotherapy. The system first suppresses metal artifacts using wavelet-based denoising and then transforms kil…
-
Transformer architecture explained: self-attention, RoPE, and FFNs
The Transformer architecture, introduced in the "Attention Is All You Need" paper, is fundamental to modern Large Language Models (LLMs). Key components include self-attention, which calculates token relationships, and …
-
CLEF foundation model advances clinical EEG interpretation
Researchers have developed CLEF, a new foundation model designed for interpreting clinical electroencephalogram (EEG) data. Unlike previous models that focus on short EEG segments, CLEF can process entire EEG sessions a…
-
Transformer大语言模型架构趋向标准化栈
对2017年至2025年间53个大语言模型的最新分析显示，Transformer架构正显著趋同。这一事实上的标准包括预归一化 (RMSNorm)、旋转位置嵌入 (RoPE)、MLP中的SwiGLU激活函数以及共享键值注意力机制 (MQA/GQA)。这种趋同归因于优化稳定性提高、每FLOP质量提升以及内核可用性和KV缓存经济性等实际考量。
-
Mela language model mimics brain memory consolidation
Researchers have introduced Mela, a novel memory-augmented language model that draws inspiration from neuroscientific theories of memory consolidation. Mela utilizes a Hierarchical Memory Module (HMM) with distinct sub-…
-
Phase-Coherent Transformer advances complex-valued neural networks
Researchers have developed a new neural network architecture called the Phase-Coherent Transformer (PCT). This model modifies the attention mechanism of standard Transformers to better preserve phase information across …
-
New Mamba-based network improves EEG decoding for stroke patients
Researchers have developed CFSPMNet, a novel framework designed to improve the decoding of motor imagery electroencephalography (MI-EEG) signals for stroke patients. This new model addresses the challenge of cross-patie…
-
New RL algorithm adaptively chunks actions for better learning
Researchers have introduced Adaptive Action Chunking (ACH), a new algorithm for reinforcement learning that dynamically adjusts the length of action sequences. Unlike previous methods that used fixed chunk lengths, ACH …
-
Transformer sentiment analysis shows link to psychotherapy patient distress
Researchers have explored Transformer-based sentiment analysis models as potential psychometric tools in psychotherapy. A study utilizing these models on a corpus of psychotherapy sessions found that aggregated sentimen…
-
LLM KV缓存详解：速度与内存的权衡
大型语言模型利用KV缓存来加速推理，通过存储先前计算出的键（key）和值（value）向量，而不是为每个新令牌重新计算它们。该技术在初始、计算密集型的“预填充”（prefill）阶段（缓存构建时）之后，显著加快了令牌生成速度。然而，KV缓存以增加内存使用量为代价来减少计算量，缓存大小随上下文长度线性增长，并且在大规模部署时可能超过模型权重。
-
NVIDIA Star Elastic embeds multiple reasoning models in one checkpoint
NVIDIA researchers have introduced Star Elastic, a novel post-training method that embeds multiple reasoning models of varying parameter sizes within a single checkpoint. This approach allows for the extraction of small…
-
LLMs Explained: Understanding Transformer Architecture and Applications
This article provides a foundational explanation of Large Language Models (LLMs), detailing their role in revolutionizing Natural Language Processing. It covers how LLMs are trained on extensive text data to understand …