ghost norm

Untied embeddings boost private LLM training accuracy and efficiency

A new study published on arXiv explores the impact of weight tying in decoder-only Large Language Models (LLMs) when fine-tuned using Differentially Private Stochastic Gradient Descent (DP-SGD). The research found that …