IntSeqBERT: Learning Arithmetic Structure in OEIS via Modulo-Spectrum Embeddings
Researchers have developed IntSeqBERT, a novel Transformer model designed to understand and predict integer sequences found in the On-Line Encyclopedia of Integer Sequences (OEIS). Unlike standard models, IntSeqBERT uses a dual-stream approach with continuous magnitude embeddings and sin/cos modulo embeddings to handle the vast range of values and arithmetic structures present in OEIS. The model achieved high accuracy in magnitude prediction and modulo prediction, significantly outperforming a tokenized Transformer baseline. A probabilistic Chinese Remainder Theorem solver further enhanced its next-term prediction capabilities, demonstrating the model's effectiveness in capturing complex arithmetic patterns. AI