static short convolutions

Dynamic convolutions boost Transformer performance in LLMs

Researchers have introduced dynamic short convolutions as a new primitive to enhance Transformer architectures used in large language models. These dynamic convolutions utilize input-dependent filters, increasing expres…