Hierarchical Residuals

Hierarchical Residuals enhance transformer efficiency by decoupling residual width

Researchers have introduced Hierarchical Residuals (HiRes), a novel architectural modification for transformers that decouples residual width from compute width. This method groups layers into blocks, using standard den…