Tokenization proven NP-complete and APX-hard, even for binary alphabets

A new research paper demonstrates that tokenization, a fundamental process in natural language processing, is computationally intractable even over bounded alphabets. The study proves that both bottom-up and direct tokenization methods are NP-complete and APX-hard, even when restricted to binary alphabets. These findings suggest that the inherent difficulty of tokenization is not due to complex constructions or large alphabets, but is a fundamental barrier, explaining the heuristic nature of current algorithms like BPE and UnigramLM and highlighting the need for approximation algorithms in future research. AI

IMPACT Establishes fundamental computational limits for tokenization, impacting the efficiency and design of future NLP models.

RANK_REASON Research paper analyzing the computational complexity of tokenization algorithms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• byte-pair encoding
• UnigramLM
• Violeta Kastreva

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →