Dominant-Layer ZO

Single LLM Layer Dominates Zeroth-Order Fine-Tuning

Researchers have discovered that fine-tuning a single layer in large language models (LLMs) can be as effective as tuning the entire model when using Zeroth-Order (ZO) optimization. This dominant layer, identified by an…