Together AI has announced their new inference optimization framework, Together Compiler, which aims to improve AI model performance. Initially, the company developed "megakernels" that fused entire model forward passes into single GPU kernels, achieving high speeds but proving difficult to productionize. Through further experimentation, Together AI found that by leveraging existing tools like CUDA Graphs and Torch Inductor, they could recover most of the megakernel performance without custom kernel writing, and even surpass previous speeds with their new compiler. AI
IMPACT Together AI's new compiler aims to optimize inference performance, potentially leading to faster and more efficient AI model deployment.
RANK_REASON The cluster describes a new product/framework release from an AI company, but not a frontier model release or significant industry-wide event.
Read on X — Together (inference / OSS) →
- CUDA Graphs
- FlashAttention
- Nvidia
- NVIDIA H100
- PyTorch
- ThunderKittens
- Together AI
- Together Compiler
- Torch Inductor
- Megakernels
- X
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →