Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP
This blog post details how to profile PyTorch code, focusing on the `nn.Linear` module and its underlying operations. It explains that `nn.Linear` wraps matrix multiplication and addition, and that PyTorch optimizes this by transposing weights on the CPU and folding the bias addition into the matrix multiplication kernel via an epilogue. The post uses an NVIDIA A100 GPU and Hugging Face infrastructure to demonstrate profiling traces. AI
IMPACT Provides insights into optimizing deep learning model performance through PyTorch profiling.