All-transistor physical-learning transformer achieves in-context learning

Researchers have developed a novel all-transistor physical-learning transformer that can perform in-context learning. This new design addresses limitations in existing equilibrium propagation systems, which struggle with complex nonlinear operations like multiplication required for attention mechanisms. By incorporating additional circuit elements and leveraging the properties of sub-threshold MOSFET devices, the simulated transformer successfully learned an in-context task, even with approximate gradient information. The training process utilized cross-entropy loss and the Lion optimizer, which requires minimal memory per parameter. AI

IMPACT This research explores novel hardware architectures for AI, potentially leading to more efficient and specialized learning systems.

RANK_REASON Academic paper detailing a novel hardware implementation for AI. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Equilibrium Propagation
• Hugging Face
• Lion optimiser
• cross-entropy loss
• MOSFET
• Multi-Layer Perceptrons
• transformers

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →