Research: LLMs show significant Chinese data pollution from tokens

A new research paper proposes a method to detect and analyze "Polluted Chinese" (PoC) tokens within Large Language Models (LLMs). These tokens, often associated with pornography or online gambling, are found to be prevalent in many LLMs, with GPT models exhibiting the worst performance, showing over 40% of long Chinese tokens falling into these categories. The study validates its speculation method on datasets like C4 and Pile, and estimates that content related to "Yui Hatano" might constitute around 0.5% of GPT-4o's training data. AI

IMPACT Highlights potential biases and harmful content in LLM training data, necessitating improved data curation and filtering.

RANK_REASON The cluster contains a research paper detailing a new methodology for analyzing LLM training data. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• C4 model
• generative pre-trained transformer
• GPT-4.1
• GPT 4.5
• GPT-4o
• GPT-4o mini
• GPT-o1
• GPT-o3
• GPT-o4-mini
• Pile
• Qingjie Zhang
• Yui Hatano

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →