IntSeqBERT Transformer Learns Arithmetic Structure in Integer Sequences

Researchers have developed IntSeqBERT, a novel Transformer model designed to understand and predict integer sequences found in the On-Line Encyclopedia of Integer Sequences (OEIS). Unlike standard models, IntSeqBERT uses a dual-stream approach with continuous magnitude embeddings and sin/cos modulo embeddings to handle the vast range of values and arithmetic structures present in OEIS. The model achieved high accuracy in magnitude prediction and modulo prediction, significantly outperforming a tokenized Transformer baseline. A probabilistic Chinese Remainder Theorem solver further enhanced its next-term prediction capabilities, demonstrating the model's effectiveness in capturing complex arithmetic patterns. AI

RANK_REASON This is a research paper detailing a new model architecture and its performance on a specific task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Chinese remainder theorem
• Hugging Face
• IntSeqBERT
• Kazuhisa Nakasho
• On-Line Encyclopedia of Integer Sequences
• transformer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →