TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
Researchers are developing new tabular foundation models (TFMs) to improve efficiency and performance. TabSwift enhances the TabPFN architecture with row-wise attention and learnable tokens for competitive accuracy and faster inference. LimiX-2M, a smaller model, also outperforms larger baselines by addressing attention bottlenecks and using a novel tokenization framework. Additionally, efforts are underway to speed up TFM pretraining through community-driven 'speedruns' and to compress datasets for faster inference and reduced memory usage. AI
IMPACT These advancements aim to make tabular foundation models more efficient and accessible, potentially accelerating their adoption in real-world applications.