Davis--Kahan

New research links optimizer choice to reduced forgetting in LLM finetuning

Researchers have explored the impact of optimizer consistency during the fine-tuning of large language models. One study suggests that using the same optimizer for both pre-training and fine-tuning leads to less knowled…