PulseAugur
EN
LIVE 11:59:12

LionVote mechanism optimizes learning rates for Lion optimizer

By PulseAugur Editorial · [2 sources] ·

Researchers have developed LionVote, a novel mechanism designed to optimize the learning rate for the Lion optimizer across different layers of neural networks. By analyzing per-layer diagnostics, LionVote identified that Lion's effective learning scale was too high for certain parameters in ViT-Tiny models trained on CIFAR-100. This new method achieved a slight improvement in top-1 accuracy compared to the standard Lion optimizer and AdamW on this specific task. AI

IMPACT Introduces a method to potentially improve training efficiency and performance for certain neural network architectures.

RANK_REASON Academic paper detailing a new optimization technique for a specific AI model. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →

LionVote mechanism optimizes learning rates for Lion optimizer

COVERAGE [2]

  1. arXiv cs.LG TIER_1 English(EN) · Kris Atallah (New York University, New York, USA) ·

    LionVote: Per-Layer Learning Rate Adaptation for Lion

    arXiv:2607.09266v1 Announce Type: new Abstract: Per-layer diagnostics reveal that, at the prescribed learning rate, Lion's effective scale is 2.6-2.8x too high for attention and MLP parameters and ~2x too high for normalization layers on ViT-Tiny/CIFAR-100; this 32% cross-layer-t…

  2. arXiv cs.LG TIER_1 English(EN) · Kris Atallah ·

    LionVote: Per-Layer Learning Rate Adaptation for Lion

    Per-layer diagnostics reveal that, at the prescribed learning rate, Lion's effective scale is 2.6-2.8x too high for attention and MLP parameters and ~2x too high for normalization layers on ViT-Tiny/CIFAR-100; this 32% cross-layer-type disparity cannot be reproduced by a single g…

RELATED ENTITIES

RELATED TOPICS