UnigramLM
PulseAugur coverage of UnigramLM — every cluster mentioning UnigramLM across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
Tokenization proven NP-complete and APX-hard, even for binary alphabets
A new research paper demonstrates that tokenization, a fundamental process in natural language processing, is computationally intractable even over bounded alphabets. The study proves that both bottom-up and direct toke…
-
New tokenizer method improves cross-lingual homograph handling in LLMs
Researchers have identified a limitation in multilingual language models where shared subword vocabularies can lead to identical surface forms being treated too uniformly across languages, even when their meanings diffe…
-
LangMAP tokenization adapts multilingual models without vocabulary changes
Researchers have developed LangMAP, a novel language-adaptive tokenization approach that extends the UnigramLM algorithm for multilingual settings. This method allows for language-specific tokenization from a single sha…
-
New ToaST tokenizer cuts token counts by over 11%
Researchers have developed a new subword tokenization method called Tokenization with Split Trees (ToaST). This method optimizes compression by recursively splitting text into binary trees and selecting vocabulary based…