Brief

A pull request to the llama.cpp project introduces a CUDA implementation of the Fast Walsh-Hadamard Transform (FWHT). This optimization, developed by user am17an, aims to speed up operations when quantizing the key-value cache. Initial benchmarks show modest performance gains, with a 1-2% boost in processing power (pp) and a 7-9% increase in token generation (tg) for the Gemma 4 26B model. AI

A developer demonstrated running large language models and image generation software on an older AMD RX 580 GPU with 8GB of VRAM, a feat previously thought impossible for such hardware. By leveraging the Vulkan backend for the ggml project, which powers tools like llama.cpp and stable-diffusion.cpp, the developer achieved a 3-4x performance increase over CPU-only processing. This approach bypasses the need for CUDA, ROCm, or DirectML, proving that modern AI tasks can be accessible on more modest, older hardware. AI

A technical guide demonstrates how to run large language models (LLMs) on older AMD RX 580 graphics cards, which were previously considered obsolete for AI tasks. The method utilizes native Vulkan, bypassing the need for CUDA or ROCm, and employs a dual-architecture approach. This involves using the GPU for smaller models via Vulkan acceleration and the CPU for larger, more demanding models, with NVMe storage identified as a critical factor for reducing model load times. AI

A developer has created TitanCore Core-1, an open-source infrastructure for training trillion-parameter LLMs. Written in C++ and CUDA, it targets VRAM limitations by implementing ZeRO-3 FSDP and fused kernels. This approach reportedly achieves a 2.6x speedup over traditional methods by optimizing memory bandwidth utilization. AI

A developer is creating a cgo-free CUDA binding for the Go programming language, aiming to simplify machine learning tool development. The project, currently in its early stages and worked on during weekends, addresses issues with large Docker images and cross-compilation inherent in cgo-based solutions. A key challenge overcome is managing CUDA's thread affinity by using a channel-based executor that locks OS threads, enabling smoother goroutine management for GPU operations. AI

The llama.cpp project has released several updates, including versions b9315, b9313, b9311, b9310, b9305, and b9301. These releases introduce various improvements and bug fixes, such as parallelizing quantization look-up table initialization and fixing checkpoint creation in the server component. The updates also provide pre-compiled binaries for a wide range of operating systems and hardware architectures, including macOS, iOS, Linux, Android, and Windows, with support for different compute backends like Vulkan, ROCm, OpenVINO, SYCL, and CUDA. AI