Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
Researchers have introduced Tensorized Engram (TN-gram), a novel memory module for large language models designed to improve how they handle multi-token patterns. Unlike previous methods that use separate memory structures for different n-gram orders, TN-gram employs shared factors in a Canonical Polyadic form. This approach allows for more efficient encoding of n-gram embeddings and has demonstrated comparable or superior performance to existing Engram modules with significantly fewer parameters. AI
IMPACT This new memory module could lead to more efficient and powerful LLMs by improving their ability to process and recall multi-token sequences.