Gelus

New research questions transformer universality, finds task-specific architectures outperform

Researchers have developed a method to optimize transformer architectures for specific datasets by replacing non-linear functions like GELUs and softmax with learned alternatives. This approach revealed that standard tr…