MathQA

New research reveals sparse and structured nature of effective LoRA writes

Researchers have found that effective Low-Rank Adaptation (LoRA) updates in language models are surprisingly sparse and structured, rather than uniformly distributed across parameters. Using a technique called Learned-B…