Causal2Vec enhances decoder-only LLMs for embeddings without architecture changes

Researchers have introduced Causal2Vec, a novel method to enhance decoder-only large language models (LLMs) for embedding tasks without altering their core architecture. This approach involves pre-encoding input text into a single 'Contextual token' which is then added to the LLM's input sequence. Causal2Vec also uses a combined embedding from Contextual and EOS tokens to mitigate recency bias, achieving state-of-the-art results on the MTEB benchmark for retrieval datasets. AI

IMPACT Introduces a new technique to improve LLM embedding performance without architectural changes, potentially reducing computational costs for specific tasks.

RANK_REASON Academic paper introducing a new method for LLM embedding models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Ailiang Lin
• BERT
• Causal2Vec
• LLM
• MTEB

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →