Similarity-Distance-Magnitude Activations
Researchers have introduced a new activation function called Similarity-Distance-Magnitude (SDM). This function aims to improve upon the standard softmax by incorporating awareness of similarity to correct predictions, distance from the training distribution, and the existing magnitude of outputs. The SDM estimator, built upon this activation, is designed to enhance interpretability and robustness against distribution shifts, particularly for selective classification tasks in pre-trained language models. AI
IMPACT Introduces a novel activation function that could improve the interpretability and robustness of large language models.