PulseAugur
EN
LIVE 11:34:37

BPE vs. Unigram-LM: Tokenization algorithms create distinct vocabularies for chemistry SMILES

By PulseAugur Editorial · [3 sources] ·

A new research paper explores the differences between two common tokenization methods, byte-pair encoding (BPE) and Unigram-LM, when applied to chemical SMILES strings. The study found that these algorithms produce significantly different vocabularies, with Unigram-LM segmenting molecules into more tokens than BPE. This indicates that the choice of subword algorithm is a critical modeling decision rather than a default setting for chemical language models. AI

IMPACT Highlights the importance of tokenization algorithm choice for chemical language models, potentially impacting downstream performance.

RANK_REASON Research paper detailing a controlled comparison of two tokenization algorithms for chemical SMILES.

Read on Hugging Face Daily Papers →

AI-generated summary · Google Gemini · from 3 sources. How we write summaries →

BPE vs. Unigram-LM: Tokenization algorithms create distinct vocabularies for chemistry SMILES

COVERAGE [3]

  1. arXiv cs.CL TIER_1 English(EN) · Hunter Heidenreich ·

    Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

    arXiv:2607.05691v1 Announce Type: new Abstract: Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is …

  2. arXiv cs.CL TIER_1 English(EN) · Hunter Heidenreich ·

    Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

    Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabulari…

  3. Hugging Face Daily Papers TIER_1 English(EN) ·

    Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES

    Byte-pair encoding and Unigram-LM create distinctly different subword vocabularies in chemical language models, with no convergence between the two approaches across diverse corpus types and vocabulary sizes.

RELATED ENTITIES

RELATED TOPICS