silx-ai/Quasar-Preview
SILX AI has released Quasar-Preview, the initial public model in its Quasar Foundation Model series. This early checkpoint showcases the Quasar architecture, featuring a sparse Mixture-of-Experts (MoE) design with approximately 18 billion total parameters and 2 billion active parameters. It incorporates a hybrid recurrent and attention layer configuration, including Loop Transformer and Quasar hybrid attention, and an experimental 5 million token context window. AI
IMPACT Demonstrates advancements in MoE and long-context architectures, potentially influencing future model development.