UnigramLM
PulseAugur coverage of UnigramLM — every cluster mentioning UnigramLM across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New research disentangles tokenization algorithms for language models
Researchers have introduced two new tokenization algorithms, BottomUpLL and TopDownComp, to disentangle the effects of optimization objectives and search procedures in language model tokenizers. By creating a 2x2 design…
-
New UniLID method uses UnigramLM for efficient language identification
Researchers have developed UniLID, a novel method for language identification that leverages the UnigramLM tokenization algorithm. This approach is efficient, requiring minimal data and compute, and supports the additio…
-
Tokenization proven NP-complete and APX-hard, even for binary alphabets
A new research paper demonstrates that tokenization, a fundamental process in natural language processing, is computationally intractable even over bounded alphabets. The study proves that both bottom-up and direct toke…
-
New tokenizer method improves cross-lingual homograph handling in LLMs
Researchers have identified a limitation in multilingual language models where shared subword vocabularies can lead to identical surface forms being treated too uniformly across languages, even when their meanings diffe…
-
LangMAP tokenization adapts multilingual models without vocabulary changes
Researchers have developed LangMAP, a novel language-adaptive tokenization approach that extends the UnigramLM algorithm for multilingual settings. This method allows for language-specific tokenization from a single sha…
-
New ToaST tokenizer cuts token counts by over 11%
Researchers have developed a new subword tokenization method called Tokenization with Split Trees (ToaST). This method optimizes compression by recursively splitting text into binary trees and selecting vocabulary based…