Goodfire has announced a private beta for their LLM training platform, Silico, which reproduces a method called RLFR. RLFR uses probes as reward signals for reinforcement learning, a technique that has drawn comparisons to the "Most Forbidden Technique" from a LessWrong post. While concerns about obfuscation are valid, the author argues that training on model internals is not inherently forbidden and should be evaluated based on specific conditions, citing research that suggests certain uses are acceptable. The key is to maintain held-out test sets that are uncorrelated with the training proxies to ensure genuine alignment. AI
IMPACT Clarifies the nuanced conditions under which using model internals for LLM training is acceptable, potentially guiding future alignment research.
RANK_REASON The item discusses a debate around a specific LLM training technique, referencing past work and opinions, rather than announcing a new model or product release.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
