New SOAP-Bubbles method enhances neural network uncertainty estimation

Researchers have introduced SOAP-Bubbles, a novel method for estimating structured weight uncertainty in neural networks. This approach adapts the SOAP optimizer by running a variational method called IVON within the eigenspace of SOAP's preconditioner. The preconditioner then transforms the diagonal estimate into a non-diagonal covariance, resulting in a method with costs comparable to SOAP and minimal disruption to existing training pipelines. Experiments show that SOAP-Bubbles (and its associated optimizer, Eigenspace-VON or EVON) can recover exact Gaussian covariance for logistic regression and achieve superior results in language model pretraining compared to current diagonal-covariance techniques. AI

IMPACT Enhances the estimation of expressive posterior distributions for deep learning at scale, potentially improving model performance and reliability.

RANK_REASON This is a research paper detailing a new method for neural networks published on arXiv. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Eigenspace-VON
• Gotit.pub
• Hugging Face
• IArxiv
• language model pretraining
• logistic regression
• Neural Networks
• ScienceCast
• SOAP optimizer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →