Unigram
PulseAugur coverage of Unigram — every cluster mentioning Unigram across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New Latent Core Tokenizer prioritizes linguistic structure over compression
Researchers have introduced the Latent Core Tokenizer (LCT), a novel language-agnostic method for creating tokenizers that prioritizes meaningful linguistic unit discovery over simple compression. Unlike traditional met…
-
AI watermarks fail forensic tests, study finds
A new study evaluating AI watermarking methods has found that current techniques are not robust enough for forensic readiness, particularly when content is paraphrased. Researchers tested three representative methods (K…
-
minbpe vs turboBPE: Faster BPE tokenization for LLMs
Two distinct implementations of the Byte-Pair Encoding (BPE) tokenizer algorithm are compared: minbpe, a pure Python educational tool, and turboBPE, a significantly faster C-extension based implementation. While minbpe …
-
New LLM Watermarking Techniques Face Evasion and Robustness Challenges
Researchers have developed several new methods to address vulnerabilities in large language model (LLM) watermarking techniques. One approach, SeedHijack, targets the pseudo-random number generator (PRNG) to manipulate …
-
New ConvexTok algorithm optimizes NLP tokenization using convex optimization
Researchers have developed a new tokenization algorithm called ConvexTok, which uses convex optimization to construct tokenizers. Unlike existing greedy methods like BPE and Unigram, ConvexTok considers the entire vocab…