New research identifies 'harmful continuation' in LLM training data

Researchers have identified a phenomenon called "harmful continuation" in the training data for large language models (LLMs) that use long chain-of-thought (CoT) traces. This occurs when reasoning in the trace continues beyond what is necessary to support the answer, and this extraneous reasoning is included in the supervised target. Experiments showed that removing this post-conclusion continuation improved LLM fine-tuning outcomes, suggesting it negatively impacts training. The study also characterized this harmful continuation by observing persistent local uncertainty and weakened directional progress, leading to the development of a proxy method called Harmful Continuation Cut (HCC) to approximate its boundary. AI

IMPACT Identifies a specific issue in LLM training data that may hinder performance and offers a method to mitigate it.

RANK_REASON The cluster contains an academic paper detailing a new finding about LLM training data. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Harmful Continuation Cut
• LLM

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →