Transformer研究深入探讨安全漏洞、训练动态和上下文学习的局限性

报道来源 [4]

1. arXiv cs.AI TIER_1 English(EN) · Jingwen Leng · 2026-05-06 13:31

   On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference

   For Transformer models, cryptographically secure inference ensures that the client learns only the final output, while the server learns nothing about the client's input. However, securely computing nonlinear layers remains a major efficiency bottleneck due to the substantial com…

2. arXiv cs.LG TIER_1 English(EN) · Yi Liu · 2026-04-28 04:00

   The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

   arXiv:2604.22778v1 Announce Type: new Abstract: We present the first systematic study of weight matrix singular value spectra \emph{during} transformer pretraining, tracking full SVD decompositions of every weight matrix at 25-step intervals across three model scales (30M--285M p…

3. arXiv cs.LG TIER_1 English(EN) · Hanna R{\o}d, Dagny Streit, Nils Valseth Selte, Justin Li · 2026-04-28 04:00

   When Context Sticks: Studying Interference in In-Context Learning

   arXiv:2604.23371v1 Announce Type: new Abstract: This paper investigates context stickiness in in-context learning (ICL), a phenomenon where earlier examples in a prompt interfere with a transformer's ability to adapt to later tasks. Using synthetic regression tasks over linear an…

4. arXiv stat.ML TIER_1 English(EN) · Soo Min Kwon, Alec S. Xu, Can Yaras, Laura Balzano, Qing Qu · 2026-04-30 04:00

   Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective

   arXiv:2505.14808v2 Announce Type: replace Abstract: The transformer's remarkable ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its strengths and limitations. However, a theoretical understanding of when ICL can and cannot …