A new benchmark called WildTrace has been developed to evaluate the effectiveness of training data detection methods in large-scale, realistic post-training scenarios for Large Language Models (LLMs). Existing evaluation methods often use benchmarks that don't reflect real-world conditions or are based on small-scale training setups, leading to potentially inaccurate performance metrics. WildTrace aims to address this by using a multi-stage pipeline that considers models throughout the post-training lifecycle, including SFT, DPO, and RL, and constructing model-specific evaluation sets with verifiable membership labels. Initial results indicate that even the best-performing detection methods achieve performance close to random guessing, highlighting the need for more robust detection techniques. AI
IMPACT Highlights the difficulty in detecting training data in LLMs, potentially impacting copyright and benchmark integrity.
RANK_REASON The item is a research paper introducing a new benchmark for evaluating LLM training data detection methods. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- DagsHub
- Direct Preference Optimization
- Gotit.pub
- Hugging Face
- IArxiv
- reinforcement learning
- ScienceCast
- supervised fine-tuning
- WILDTRACE
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
