New Riemannian Language Models achieve 2x perplexity improvement

Researchers have introduced Riemannian Language Models (RiLM), a novel approach to parameter-efficient language modeling that eliminates the need for an output matrix. This method leverages geodesic decoding, where context unfolds as a trajectory on a Riemannian manifold, and next-token probabilities are determined by the geodesic distance between the current state and vocabulary embeddings. In experiments on WikiText-2, the hyperbolic variant (HypRiLM) achieved a perplexity of 54.2, roughly doubling the performance of tied recurrent baselines and significantly outperforming a flat manifold implementation. AI

IMPACT This research could lead to more efficient language models for edge deployment and domain adaptation by reducing parameter count.

RANK_REASON The cluster describes a new research paper introducing a novel language modeling technique. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Flat RiLM
• geodesic decoding
• Hugging Face
• HypRiLM
• long short-term memory
• Penn Treebank
• Riemannian Language Models
• State Space Model
• Transformer++
• WikiText-2

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →