Press & Wolf

Weight tying in language models: Gradient analysis and performance impact

Researchers explored the implications of weight tying in language models, specifically how tying the input embedding and output projection matrices affects gradient calculations and model performance. They found that de…