A user on the r/MachineLearning subreddit is seeking information about the successful application of Exponential Moving Average (EMA) techniques specifically with LoRA (Low-Rank Adaptation) adapters. They are interested in scenarios where the EMA adapter functions as a self-teacher, generating soft labels for the trainable adapter. The user references a paper on on-policy self-distillation that uses EMA for the teacher but involves full fine-tuning, and is looking for empirical results demonstrating this concept working with LoRA or similar parameter-efficient fine-tuning methods. AI
IMPACT This query highlights a specific area of interest in efficient model fine-tuning techniques.
RANK_REASON User query on a technical topic within a subreddit.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
