A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
Researchers have identified a boundary-layer mechanism that explains a one-third scaling in online softmax classification. This mechanism shows that only examples near the teacher's decision boundaries contribute significantly to learning at later stages. The study predicts a power-law learning curve of \(\\alpha^{-1/3}\\) for test loss and generalization error, which is slower than the Bayes-optimal reference. They also suggest that learning-rate schedules can improve generalization error towards a \(\\alpha^{-1/2}\\) power law. AI
IMPACT Identifies a theoretical limitation in current classification methods and suggests potential improvements through learning-rate adjustments.