Brief

Two new research papers explore the limitations and advantages of large language models. One paper argues that even with abundant data, there are fundamental limits to adaptation in multitask learning, suggesting that simply increasing data size won't overcome these challenges. The second paper investigates why larger models perform better, attributing their success to a reduced interference mechanism that allows them to retain information on rare and complex tasks, a feat smaller models struggle with. AI

Three new research papers explore the concept of "grokking" in machine learning, specifically within the context of ridge regression. One paper presents a numerical procedure to find optimal regularization strength, demonstrating near-optimal generalization. Another paper provides theoretical proofs for grokking in linear models trained with gradient descent and weight decay, suggesting it's a training condition rather than a fundamental flaw. The third paper connects stochastic resetting from physics to ridge regularization, showing how resetting to the origin can replicate the ridge estimator and exploring alternative spectral filters with different renewal laws. AI