Attention, not scale, drives human-AI alignment in multimodal language prediction
A new study published on arXiv suggests that the attention mechanisms within transformer models, rather than their sheer scale, are the primary drivers of alignment with human behavior in multimodal language prediction. Researchers found that adding visual context significantly improved model-human alignment in predicting words, with transformer attention maps correlating with human gaze patterns. This indicates that current vision-language models can effectively leverage visual cues to approximate human language prediction, highlighting the importance of selective attention over model size. AI
IMPACT Highlights that attention mechanisms, not just model size, are key to aligning AI with human language prediction using visual context.