KID (Knowing / Intent / Doing)

New KID framework challenges role assignment for transformer attention heads

Researchers have demonstrated that common methods for assigning specific roles to attention heads in transformer models are insufficient. Their study, involving three instruction-tuned models, found that heads identifie…