Researchers have explored token sparsification techniques to improve the efficiency of audio transformers, which are computationally intensive due to self-attention mechanisms. While token sparsification has been studied in vision and language, its application in audio is less developed. This work provides a broad evaluation of various token pruning and merging strategies across different datasets and model sizes, considering both post-training and fine-tuning scenarios. The study introduces a new method that integrates attention scores with spectral features from token embeddings, demonstrating that frequency-aware approaches, particularly those using spectral flatness and low-frequency energy, outperform attention-only methods. Additionally, sparsification based on spectrogram patch intensity proves effective for longer audio inputs. AI
IMPACT This research could lead to more efficient audio processing models, enabling wider deployment on resource-constrained devices.
RANK_REASON Academic paper detailing novel methods for improving transformer efficiency. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
