Scaling Hypothesis #2: Are Humans Just More Over-Parameterized?
A new hypothesis suggests that the differences between human intelligence and current deep learning models, particularly LLMs, stem from a bias-variance tradeoff. The proposal posits that human brains minimize bias through overparameterization and high-learning-rate training on diverse datasets, leading to efficient generalization. Conversely, LLMs are theorized to minimize variance, resulting in strong performance but limited generalization and memorization. This 'catapulted LLM' approach could enhance generalization, improve adversarial robustness, and provide a more stable foundation for AI safety. AI
IMPACT This hypothesis could lead to more generalizable and safer AI models by reframing training strategies.