Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
Researchers have developed a new framework called Item Response Scaling Laws (IRSL) that integrates Item Response Theory with language model scaling laws. This approach aims to make the estimation of scaling laws more efficient and generalizable by disentangling model ability from question characteristics, reducing the complexity from O(M x N) to O(M + N). IRSL uses empirical response probabilities from LMs, such as token probabilities or pass rates, to derive more reliable scaling estimates with significantly fewer questions, enabling accurate performance forecasting across different benchmarks. AI
IMPACT This framework could significantly reduce the computational cost of evaluating and forecasting AI model performance.