New Mixture of Channel Experts layer slashes CNN computation

Researchers have introduced Mixture of Channel Experts (MoCE), a novel layer designed to replace standard pointwise projections in convolutional neural networks. Unlike traditional Mixture-of-Experts (MoE) models that struggle with parallel convolutional experts learning similar filters, MoCE shifts the expert axis to channel selection. This approach uses a single output channel as an expert, with a learned sparse support of input channels, and an input-adaptive softmax for aggregation. MoCE significantly reduces computational cost and latency while matching or exceeding the performance of dense baselines on benchmarks like ImageNet-1K and CIFAR-100. AI

IMPACT Introduces a more efficient method for convolutional neural networks, potentially reducing computational costs and latency in vision models.

RANK_REASON This is a research paper detailing a new method for convolutional neural networks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• CIFAR-100
• EfficientViT
• ImageNet-1K
• Mixture of Channel Experts
• Mixture-of-Experts
• pointwise projections

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →