Multi-Granular Node Pruning for Causal Circuit Discovery
Researchers have developed a novel node-level pruning framework for discovering causal circuits within large language models (LLMs). This method allows for more granular identification of essential subnetworks, down to individual neurons, overcoming the limitations of existing edge-pruning techniques that focus on coarser units like attention heads or MLP blocks. The framework uses learnable masks and granularity-specific sparsity penalties to achieve comprehensive compression in a single fine-tuning run, resulting in smaller discovered circuits and a significantly lower memory footprint compared to prior methods. AI
IMPACT This research offers a more efficient and granular method for understanding the internal workings of LLMs, potentially aiding in interpretability and targeted model improvements.