Brief

A new research paper proposes that the computational and memory bottlenecks in large language models (LLMs) related to attention mechanisms are artificial and can be overcome through principled sparsity. The study, which analyzed 20 models across five families, found that current LLMs are surprisingly robust to inference-time decode sparsity, even without specific training for it. This approach could significantly accelerate LLM inference, with sparse decode kernels achieving up to 10x speedups on hardware like the H100 at 50x sparsity levels. AI

Photoroom significantly reduced its image generation costs by optimizing its diffusion pipeline. The company achieved a 39% cost reduction on the UNet denoising stage through int8 quantization and a 79% reduction in text-encoder costs by caching LLM embeddings. Implementing an AI gateway with Bifrost further decreased caption API spend by 61% and improved latency, while also mitigating costs associated with upstream LLM outages. AI

Single-image diffusion model inference is slowed by kernel launch overhead and attention memory traffic, rather than raw computational power. Optimizing with `torch.compile` in `reduce-overhead` mode, employing a fused attention backend, and batching classifier-free guidance can significantly reduce latency. Only after these optimizations should one consider distillation methods for further speed improvements, while carefully evaluating potential quality degradation. AI

This guide details how to run advanced large language models locally on personal hardware in 2026, bypassing expensive API costs. It emphasizes that VRAM is the primary hardware bottleneck, not raw compute power, and suggests specific GPU configurations for different budgets. The guide recommends using Ollama as the standard tool for managing local LLMs and highlights several Chinese models, such as Qwen 2.5 and DeepSeek-R1, for their strong performance relative to their size. AI

Researchers have developed a new metric called Overall FLOP Utilization (OFU) to measure GPU efficiency for AI workloads. OFU is derived from on-chip performance counters and does not require application instrumentation, making it applicable across different GPU generations and precisions. When tested on production training jobs, OFU showed a strong correlation with application-level metrics and helped identify efficiency regressions and framework miscalculations. AI