Canonicalized Stable-List Replay for Private Federated Continual Learning over Language-Model Embeddings
Researchers have developed a new method called Canonicalized Stable-List Replay (CSLR) to improve private federated continual learning for language models. CSLR addresses the challenge of clients sharing only small, noisy, and unordered lists of replay summaries under differential privacy. By using public anchor sentences to align candidate replay distributions in a shared embedding space, CSLR enhances identifiability for aggregation. Experiments show CSLR improves performance on continual classification, NER, and dialogue tasks by 3.9-5.6 points over existing baselines. AI
IMPACT Introduces a novel technique to improve privacy and performance in federated learning for language models, potentially enabling more robust on-device adaptation.