Researchers have introduced ZOIF, a novel framework for estimating influence functions in large language models (LLMs) that bypasses the need for backpropagation. This method utilizes only forward passes, significantly reducing memory requirements and making data selection more feasible for LLMs. ZOIF decomposes the process into three stages, including approximating gradients with zeroth-order estimation and solving inverse Hessian-vector products via optimization, ultimately yielding per-sample influence scores. Experiments demonstrate that ZOIF can select data subsets with utility comparable to or better than existing methods, while consuming substantially less peak memory and similar computation time. AI
IMPACT Enables more efficient and memory-conscious data selection for LLMs, potentially improving model training and debugging.
RANK_REASON The cluster contains a research paper detailing a new method for LLM data selection. [lever_c_demoted from research: ic=1 ai=1.0]
- arXiv
- backpropagation
- finite difference
- Hessian matrices
- Hugging Face
- Influence Functions
- LLM
- mathematical optimization
- zeroth-order estimation
- ZOIF
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
