Understanding Wacky Weights: A Dissection of SPLADE's Learned Term Importance
Researchers have conducted a systematic investigation into "wacky weights" within SPLADE, a learned sparse retrieval model. These weights, which assign importance to expansion terms that seem semantically unrelated to the input, can limit the model's interpretability. The study found that larger vocabularies correlate with a higher prevalence of these wacky tokens, while stricter sparsity regularizers reduce their occurrence. The research indicates that these weights are primarily used for in-domain effectiveness rather than out-of-domain generalization. AI
IMPACT Provides a deeper understanding of interpretability challenges in sparse retrieval models, potentially guiding future research in explainable AI.