Lngram: N-gram Conditional Memory in Latent Space
Researchers have introduced Lngram, a novel module for sequence modeling that operates in latent space. Unlike previous methods that rely on tokenization, Lngram learns discrete symbols directly from hidden states and performs N-gram lookups. This approach has demonstrated improved performance in long-context language modeling and effectively incorporates domain knowledge when added to pre-trained models. The module also shows promise in vision-language and vision-language-action tasks, suggesting broader applicability beyond text. AI
IMPACT Introduces a new method for sequence modeling that could improve performance and efficiency in various AI tasks.