Brief

Researchers have found that specific training techniques can encourage spatial locality in Vision Transformers. By using a 'Modern' protocol involving data augmentation like CutMix and ColorJitter, along with label smoothing, early layers of ViTs showed more concentrated attention patterns. An ablation study revealed that CutMix was the primary driver of this effect, significantly reducing the Mean Attention Distance compared to baseline methods. AI

Researchers have developed FAIR-Pruner, a new framework designed for automatic, layer-wise structured pruning of deep neural networks. This method adaptively allocates sparsity across network layers by using both removal-oriented and protection-oriented signals. Experiments across various datasets and model architectures, including vision models and a Qwen1.5-MoE model, demonstrate that FAIR-Pruner achieves strong accuracy-compression trade-offs. The framework is available as an open-source package. AI

Researchers have developed AutoMCU, a novel system that leverages LLM-based multi-agent approaches to customize neural networks for microcontroller units (MCUs). This method prioritizes feasibility by integrating vendor toolchain feedback early in the design process, significantly reducing the search cost and time compared to traditional hardware-aware neural architecture search methods. AutoMCU has demonstrated competitive accuracy on benchmark datasets and successful deployment on STM32 microcontrollers, making edge intelligence more accessible. AI

A new research paper investigates how the allocation of sparsity in neural networks impacts their ability to recover accuracy after pruning, especially when labeled retraining data is unavailable. The study compares different sparsity allocation methods like ERK and LAMP across various datasets and architectures, finding that the choice of allocation significantly affects post-repair accuracy. Researchers identified a critical transition regime where standard repair methods begin to fail, highlighting the need to jointly consider pruning allocation and repair strategies. AI

Researchers have developed a new adversarial attack method called Mixed Dynamic Spiking Estimation (MDSE) specifically for Spiking Neural Networks (SNNs). This attack demonstrates that the effectiveness of white-box adversarial attacks on SNNs is heavily influenced by the choice of surrogate gradient estimator. The MDSE attack is designed to exploit multiple surrogate gradient estimators simultaneously, enabling it to generate adversarial examples that can fool both SNNs and traditional non-SNN models like Vision Transformers and CNNs. AI

Researchers have developed a new method called Early High-Frequency Injection (EIHF) to improve out-of-distribution (OOD) detection in computer vision models. EIHF works by injecting high-frequency information into the input data before it's processed by the first convolution layer, without altering the training objective. This approach enhances the model's ability to distinguish between in-distribution and out-of-distribution data, particularly for geometry-sensitive tasks, by reshaping feature geometry and reducing overlap in scores. Experiments on CIFAR-100 and ImageNet-100 datasets showed promising results, including improved false positive rates and area under the receiver operating characteristic curve. AI

Researchers have introduced HamJEPA, a novel approach to Joint Embedding Predictive Architectures (JEPAs) that moves beyond isotropic regularization. This new method encodes views as phase-space states and uses a learned Hamiltonian leapfrog map for cross-view prediction. Experiments on CIFAR-100 and ImageNet-100 show significant improvements in kNN and linear probe accuracy compared to existing methods like SIGReg. AI

Researchers have introduced SpikingMoE, a novel framework that combines Spiking Neural Networks (SNNs) with a Mixture-of-Experts (MoE) architecture. This approach utilizes a spike-driven prompt (SDprompt) for biologically plausible, input-dependent routing of information to different expert modules. Designed for neuromorphic hardware, SpikingMoE aims to enhance energy efficiency in visual recognition tasks while maintaining competitive performance, achieving high accuracy on CIFAR-10 and CIFAR-100 datasets. AI

Researchers have developed a novel method called Code-Oriented LM Embeddings (COLE) to improve Neural Architecture Search (NAS). This technique uses off-the-shelf language models to generate embeddings from code representations of neural architectures, bypassing the need for expensive fine-tuning or complex feature engineering. Experiments on NAS-Bench-201 and einspace demonstrated that COLE embeddings outperform other text-based encodings and significantly reduce the evaluation budget required to find high-performing architectures. AI