L40S
PulseAugur coverage of L40S — every cluster mentioning L40S across labs, papers, and developer communities, ranked by signal.
8 day(s) with sentiment data
L40S GPU startup time impacted by model download speed
The L40S GPU's startup time is significantly affected by the speed of downloading models from sources like Hugging Face. A 30-second addition to startup was observed due to model download, with local caching reducing this by over half. This highlights that storage I/O can be a bottleneck for initial GPU utilization in certain workflows.
Unsloth library may enable L40S for fine-tuning larger models than previously feasible
Given Unsloth's reported ability to reduce VRAM needs by up to 70% and enable fine-tuning of 70B models on consumer GPUs, it's plausible that the L40S, a more powerful card than typical consumer GPUs, could be used for fine-tuning even larger models or with larger batch sizes than currently benchmarked, potentially rivaling higher-end enterprise cards for specific tasks.
QLoRA performance anomaly observed on L40S
A recent benchmark on an L40S GPU indicated that QLoRA was slower and used more memory than standard LoRA when fine-tuning the Qwen2.5-1.5B-Instruct model. This contradicts typical expectations of QLoRA's efficiency and suggests potential implementation-specific issues or specific model/dataset interactions that warrant further investigation on the L40S.
-
Self-hosting LLMs to cost $150K-$2M+ annually in 2026
Self-hosting open-source large language models in 2026 is projected to cost between $150,000 and over $2 million annually. The significant expenses extend beyond GPU hardware to include the cost of specialized engineeri…
-
Reddit users debate hardware for local AI model inference
A user on the r/MachineLearning subreddit is seeking advice on the best hardware for running local AI models, specifically asking for recommendations between a MacBook Pro and a Mac Mini. The user has access to powerful…
-
Hugging Face model download adds 30s to GPU startup time
A test on a Runcrate L40S GPU revealed that downloading a 7B model from Hugging Face adds approximately 30 seconds to the initial startup time. When the model weights were already cached locally, the time to first token…
-
Unsloth cuts LLM fine-tuning VRAM needs, enabling cheaper GPU use
Unsloth, an open-source library, significantly reduces the VRAM requirements for fine-tuning large language models, enabling the use of smaller, more cost-effective GPUs. By leveraging techniques like 4-bit QLoRA, Unslo…
-
LoRA outperforms QLoRA and full fine-tuning in speed and memory usage
A recent benchmark on an L40S GPU compared three fine-tuning methods for the Qwen2.5-1.5B-Instruct model: full fine-tuning, LoRA, and QLoRA. Full fine-tuning required the most memory, consuming 36.9 GiB with a micro-bat…
-
vLLM prefix caching boosts LLM throughput 7.5x, slashes latency
A recent benchmark on a single L40S GPU demonstrated that vLLM's prefix caching feature can significantly boost throughput and reduce latency for LLM inference. When tested with a 4,000-token system prompt and 64 concur…
-
Qwen2.5-7B-Instruct performance benchmarked on NVIDIA GPUs
A performance analysis of the Qwen2.5-7B-Instruct model was conducted across three NVIDIA GPUs: RTX A6000, L40S, and H100, using the vLLM framework. The study focused on tokens per second and the cost per million tokens…
-
LLM FinOps strategy cuts batch reasoning costs by 75% using UTC off-peak arbitrage
A new strategy for optimizing Large Language Model (LLM) operational costs, known as Time-Shifted LLM FinOps, has been proposed. This method involves scheduling batch reasoning tasks during off-peak UTC hours to leverag…
-
RTX 3090 vs L40S: Batch size dictates GPU performance and cost
A performance comparison between the RTX 3090 and L40S GPUs reveals that the 3090 is faster at batch size 1, outperforming the L40S by up to 1.11x. However, the L40S significantly pulls ahead at larger batch sizes, reac…
-
3.8B LLM trained for under $1,000, challenging hyperscaler dominance
A 3.8 billion parameter LLM, named Little LM, has been trained to a CORE metric of 0.384 for under $1,000. This achievement challenges the notion that only large-scale, multi-million dollar operations can pre-train comp…
-
NVIDIA GPU Guide: H100 for LLM Training, L40S for GenAI
When selecting hardware for machine learning projects, consider specific GPU models based on the task rather than defaulting to the most expensive options. The NVIDIA H100 is recommended for large language model trainin…
-
Towards AI launches enterprise division, details practical LLM builds
Towards AI has launched a new enterprise division called Towards AI Deployment, focusing on deploying AI solutions for PE-backed companies and investors. The publication also highlights practical AI applications, includ…
-
vLLM optimizations on L40S: Batching and FP8 yield major gains
A detailed analysis of vLLM optimizations on NVIDIA L40S GPUs, using Llama 3.1 8B Instruct, reveals that continuous batching is the most significant performance enhancer, offering a 73x throughput increase and substanti…
-
HexGrid Cloud offers custom LLM GPU benchmarking for open-weight models
HexGrid Cloud is offering to benchmark open-weight LLMs on user-specified GPUs and configurations. They are seeking suggestions for models and hardware setups to test their deployment platform, focusing on chat/instruct…
-
Machine0.io launches persistent VMs with CLI control
Machine0.io has launched a new service offering persistent virtual machines (VMs) for developers and agents, accessible via a command-line interface (CLI). These VMs run NixOS or Ubuntu with pre-installed tools, providi…
-
AutoMegaKernel compiles Llama models into single CUDA kernels
Researchers have developed AutoMegaKernel (AMK), a system that compiles HuggingFace Llama-family models into a single, persistent CUDA kernel for efficient forward passes. AMK's static validator ensures schedule safety,…
-
AI inference latency limited by more than memory bandwidth, study finds
A new paper reveals that the inference performance of physical AI systems, such as robots and autonomous vehicles, is not solely limited by memory bandwidth as previously assumed. The research demonstrates that while ba…
-
Idle GPU power cost driven by CUDA context, not VRAM
Researchers have quantified the energy cost of keeping AI models loaded on GPUs, a practice known as "model parking." Their study found that the primary energy drain comes from the CUDA context, which adds 26-66W of idl…
-
INT8 quantization can slow down AI inference, study finds
A recent analysis explored the performance of INT8 quantization versus FP16 precision on NVIDIA's Ada Lovelace architecture, specifically using an L40S datacenter GPU and an RTX 4090 consumer card. The findings indicate…