The RTX 5090 did not suddenly receive more CUDA cores. Its 32GB of GDDR7 did not get wider overnight, and NVIDIA did not quietly replace anyone’s card with a new revision. Yet this week NVIDIA said local inference through llama.cpp can deliver up to 1.9 times higher throughput on the same GeForce RTX 5090 after a new round of software optimization.
That number should make anyone shopping for local AI hardware slightly uncomfortable. The easiest way to make a local model faster has always sounded obvious: buy a faster GPU. Move from an older card to Blackwell, buy more memory, spend more money, and watch the token counter climb. llama.cpp is showing why that assumption is incomplete.
Access without medium partner: RTX 5090 Up to 1.9× Faster for Free
The hardware matters enormously, but the software deciding how efficiently that hardware is used can leave a surprising amount of performance untouched. In local AI, we are still early enough that a kernel change, a better decoding strategy, or less wasteful movement between CPU and GPU can sometimes feel like a hardware generation upgrade.
The important word in NVIDIA’s announcement, however, is up to. This is not a promise that every RTX 5090 owner will open LM Studio tomorrow and see every model run 90 percent faster. The gain covers a mixture of improvements across inference, and the result depends on the model, workload, prompt length, decoding method, runtime settings, and whether the optimized code path applies at all.
That makes the story better, not worse. The real question is not whether one benchmark doubled. It is why a $1,999 class GPU still had enough software headroom for a number this large.
The RTX 5090 did not change. The path through it did.
At IFA 2026, NVIDIA announced a set of local AI improvements developed with the open source llama.cpp and vLLM communities. For llama.cpp on a GeForce RTX 5090, NVIDIA says the work can deliver up to 1.9 times higher throughput through a combination of kernel optimizations, improved speculative decoding, and faster prefill.
The changes are not being kept inside an NVIDIA demo application. NVIDIA says the llama.cpp gains are available through the inference backend itself and can also be experienced through applications including LM Studio and Ollama. That matters because those are the programs many local AI users actually touch.
There is no new GPU in that sentence. There is no extra VRAM, no overclock, and no second graphics card. The improvement comes from making the workload travel through the existing hardware more efficiently.
That is exactly the kind of progress local AI needs. For the last two years, hardware discussions have been dominated by capacity. How much VRAM does the card have? Can it fit a 70B model? Does unified memory make a mini PC more useful than a faster discrete GPU? Those questions still matter, but once a model fits, software efficiency becomes the next battlefield.
A badly used RTX 5090 is still a badly used RTX 5090.
“1.9× faster” is not one thing
LLM inference looks simple from the outside. You type a prompt, the model pauses for a moment, and words begin appearing. Underneath that experience are several workloads with very different performance characteristics.
The first major phase is prompt processing, often called prefill. The system processes the input tokens you supplied, builds the internal state needed for generation, and prepares the KV cache. Long prompts, large codebases, documents, and agent histories can make this phase expensive.
Then comes decode. The model generates tokens one after another. For large models, this phase is often heavily limited by how quickly weights and state can move through memory rather than by the headline number of floating point operations printed on a GPU box.
Throughput can also mean different things depending on the test. A single person chatting with one model cares about latency and tokens per second for one sequence. A local server handling many simultaneous requests cares about total tokens produced across all of them. An agent running several subagents may care more about aggregate throughput than the speed of one response.
NVIDIA’s 1.9 times figure should therefore be read as a best case result across the optimized llama.cpp path, not as a universal multiplier for every model and every use case. That distinction is worth keeping because software benchmarks become misleading very quickly when the word “throughput” is turned into “everything is 1.9 times faster.”
Kernel optimization is the least glamorous part and probably the most important
A modern GPU spends its life running kernels, small programs that perform the operations required by the model. If those kernels move data inefficiently, launch too much work, synchronize unnecessarily, or fail to use the architecture well, expensive hardware can spend time waiting instead of calculating.
llama.cpp has been steadily improving its CUDA backend for years. The project includes custom CUDA kernels for NVIDIA GPUs and supports aggressive quantization so models can fit into smaller memory pools. That makes it one of the most important pieces of software in the local LLM world.
Earlier llama.cpp optimization work already showed how much room existed. In a November 2025 technical discussion, developers demonstrated large token generation gains on RTX 4090 and RTX 5090 systems from kernel fusion and concurrent streams. On one RTX 5090 test, Qwen3 30B A3B in Q4_K_M increased from roughly 247 tokens per second with key optimizations disabled to about 352 tokens per second with them enabled.
That was not a new graphics card. It was better execution.
The basic idea behind kernel fusion is easy to understand. If several small GPU operations can be combined, the runtime can reduce launch overhead and unnecessary intermediate memory movement. Each individual saving may look tiny, but autoregressive inference repeats similar work token after token. Small inefficiencies become visible when multiplied thousands of times.
Blackwell is extremely fast. That also means software inefficiency can become easier to notice. When the GPU can finish useful work quickly, time spent waiting on the CPU, copying data, synchronizing, or launching poorly organized kernels becomes a larger part of the total experience.
Buying faster hardware does not make software overhead disappear. Sometimes it exposes it.
Speculative decoding attacks one of LLM generation’s strangest inefficiencies
Normal autoregressive generation is painfully sequential. The model predicts a token, then uses that token to predict the next one, then repeats the process again. Even a monster GPU cannot simply generate the whole paragraph in one pass because every token depends on what came before it.
Speculative decoding tries to cheat that process without changing the final model. A smaller or cheaper draft path predicts several likely tokens ahead. The main model then verifies those predictions together, keeping the accepted tokens and rejecting the ones that do not match.
When the guesses are good, the expensive model can move forward by several tokens in a single verification step. When the guesses are bad, the benefit shrinks because the target model has to discard more of the draft.
llama.cpp now supports multiple speculative approaches, including draft models and EAGLE style mechanisms. NVIDIA specifically points to enhanced speculative decoding as one part of its new performance claim.
This is one reason the 1.9 times number cannot apply equally to every workload. Speculative decoding depends heavily on how predictable the generation is, how good the draft is, how much extra VRAM it consumes, and how efficiently the runtime verifies the predictions.
Code completion can behave differently from creative writing. A repetitive structured response can behave differently from difficult reasoning. A draft model that works beautifully beside one target model may be a poor match for another.
The speedup is real when the conditions line up. The conditions matter.
Faster prefill may be the part agent users notice first
Local AI is no longer only a person typing short questions into a chat box. Agents are increasingly loading code repositories, documents, tool descriptions, memory, previous task state, search results, and long conversations into context.
That shifts attention toward prefill. If an agent has to process 50,000 or 100,000 tokens before it starts doing useful work, generation speed is only part of the experience. A system that writes at 100 tokens per second can still feel slow if it spends an irritating amount of time digesting the prompt before the first token appears.
NVIDIA says faster prefill contributes to the new llama.cpp gains. That may matter more to agentic workloads than another small increase in ordinary chat generation.
This is also why the local AI benchmark conversation needs to grow up. A single tokens per second number is increasingly inadequate. We need prompt processing speed, time to first token, decode speed, memory usage, maximum usable context, power draw, and concurrency to understand how a machine actually feels.
The fastest number in a benchmark table may not be the delay the user is waiting on.
A community RTX 5090 test shows how silly software bottlenecks can become
A recent llama.cpp issue provides a good example of software leaving expensive hardware idle. A user benchmarking llama-server on an RTX 5090 with 32 simultaneous slots found generation throughput plateauing around 706 tokens per second. Profiling showed the server moving a large logits matrix from the GPU back to the host CPU on every decode step while the CPU then performed sampling work.
Using llama.cpp’s experimental backend sampling option moved that part of the workflow away from the old CPU path. In the user’s test, throughput increased from roughly 706 to 1,046 tokens per second, an improvement of about 48 percent, while time per output token dropped significantly.
That result is not NVIDIA’s 1.9 times benchmark, and it should not be presented as one. It is a separate community test with a specific high concurrency configuration.
It is still useful because it shows the same underlying problem. The RTX 5090 was not out of compute. The pipeline was wasting time moving information to the wrong place.
That is the kind of bottleneck people rarely think about when comparing GPUs by CUDA core count.
This is why LM Studio and Ollama support matters
Most local AI users are not compiling llama.cpp from source every morning. They are opening LM Studio, running Ollama, downloading a quantized model, and expecting the software to make sensible decisions.
That means backend optimization has an unusually large reach. When a tool such as llama.cpp improves and those changes flow into applications sitting above it, thousands or millions of users can benefit without learning the kernel that changed.
This is one of the best things about the local AI software stack. Performance work in an open source backend can spread into polished desktop applications, developer tools, agent systems, and other runtimes.
It is also why benchmark articles age so quickly. A test that declared one GPU disappointingly slow six months ago may no longer describe the same software stack today. The silicon did not change, but the runtime did.
Anyone comparing local AI hardware should therefore care about the date of the benchmark almost as much as the model being tested.
The upgrade treadmill is starting to look a little silly
The AI hardware market has trained buyers to think in generations. RTX 3090 is old. RTX 4090 is newer. RTX 5090 is the current monster. The next card will be better again.
That logic is not wrong. New GPUs bring more compute, faster memory, better low precision support, larger memory pools in some segments, and newer architectural features.
The problem is treating hardware as the only place performance comes from. If software optimization can produce a 20 percent gain, a 40 percent gain, or in a carefully chosen case approach NVIDIA’s claimed 1.9 times uplift, then the useful life of existing hardware becomes longer than spec sheet culture suggests.
A person who already owns an RTX 5090 obviously does not need another GPU to access these improvements. The more interesting case is someone looking at an expensive upgrade from a 4090, 3090, or workstation card. Before spending thousands, it may be worth asking whether the model, runtime, quantization, and backend are already using the current GPU efficiently.
Sometimes the bottleneck is hardware. Sometimes the bottleneck is the software path nobody bothered to profile.
This does not solve the RTX 5090’s biggest local AI limitation
There is one thing no llama.cpp optimization can do: turn 32GB into 64GB.
The RTX 5090 remains a 32GB GPU. If the model and its runtime state do not fit, a kernel optimization cannot repeal capacity. You can quantize more aggressively, offload layers to system RAM, use multiple GPUs, or choose a smaller model, but each option changes either performance, quality, complexity, or all three.
This is why 128GB and 192GB unified memory systems remain interesting even when they are slower on models that fit inside a 5090. Capacity and speed solve different problems.
The new software work makes the 5090 faster at the workloads it can hold. It does not make it a giant memory machine.
That distinction matters for anyone tempted to interpret 1.9 times higher throughput as a universal local AI upgrade.
New optimizations can also create new edge cases
Another reason to resist the marketing version of the story is that inference software is moving quickly. Recent llama.cpp issue reports show exactly what that looks like.
One user testing Qwen3.5 MTP speculative decoding reported a much smaller gain on an RTX 5090 than on a 4090 in a specific configuration. Another issue described a Windows build where speculative decoding collapsed to extremely low performance under certain tensor split conditions. A separate multi GPU issue involved long prompt prefill and MTP behavior.
Those reports do not disprove NVIDIA’s result. They show why “up to” exists.
Inference software has many code paths. Operating systems differ. Model architectures differ. CUDA builds differ. Context lengths differ. Quantization formats differ. A performance feature that helps one setup can expose a bug or offer little benefit in another.
The correct response is not cynicism. It is benchmarking.
What I would test before believing the 1.9× number on my own machine
I would keep the test boring and repeatable. Use the same model file, same quantization, same prompt, same context size, same sampling settings, and same operating system before and after the update. Then separate prompt processing from token generation so a huge prefill improvement does not get mistaken for a huge generation improvement.
For an agent workload, I would also test concurrency. Run several independent requests rather than one chat, because aggregate throughput can improve very differently from single user latency. Finally, watch VRAM because speculative decoding and larger contexts can consume additional memory, and an optimization that is brilliant on a 20GB model may become irrelevant if it pushes a 31GB workload out of a 32GB card.
That is the kind of benchmark I want to see from independent testers over the next few weeks. Not one giant “1.9×” bar, but a map of where the 1.9× actually comes from.
The real story is that local AI software is still immature enough to get much faster
This is the part I find more interesting than the benchmark itself. PC graphics went through decades of optimization. Game engines, drivers, APIs, compilers, and GPU architectures learned how to work together through years of painful tuning.
Local generative AI does not have that history. llama.cpp only became one of the defining local inference projects in the last few years. Quantization methods keep changing, model architectures keep changing, speculative decoding keeps changing, attention kernels keep changing, GPU architectures keep changing, and agent workloads are forcing runtimes to care about long context and concurrency in ways simple chat benchmarks did not.
There is still low hanging fruit. That should change how we think about hardware purchases.
The RTX 5090 is not merely a fixed performance number printed on launch day. Its useful AI performance can improve as the software stack learns how to use Blackwell better.
The same is true for older cards. A 3090 did not become interesting again because its silicon changed. It became interesting because 24GB of VRAM found new workloads and better software. A 4090 can gain from new kernels and runtimes long after NVIDIA stops putting it on launch slides.
Local AI hardware has a software defined lifespan. That is good news for everyone except the part of us that likes buying new GPUs.
Maybe the next local AI upgrade should be an update button
NVIDIA’s up to 1.9 times llama.cpp result should not be turned into a promise that every RTX 5090 just doubled in speed. That would be the same benchmark abuse the hardware industry has trained us to expect.
The better conclusion is more useful. Modern GPUs are fast enough that the software stack can become the bottleneck. Kernel organization matters. CPU to GPU transfers matter. Prompt processing matters. Draft token acceptance matters. Scheduling matters. A local AI runtime is not simply a thin wrapper around fixed hardware performance.
That means a person chasing faster local inference has more options than buying another graphics card. Update the runtime, check whether the application includes the new backend, test the model again, and profile what is actually slow before deciding whether hardware is still the problem.
The RTX 5090 did not become a different GPU this week. llama.cpp simply got better at using the one people already bought.
If software can still find gains approaching 1.9 times in 2026, the most interesting question may not be how fast the next GPU will be. It may be how much performance is still hiding inside the GPUs already sitting on our desks.
Sources
NVIDIA, “Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026,” September 3, 2026
https://blogs.nvidia.com/blog/local-ai-ifa-next-gen-agents-nv-pair-rtx-spark/
llama.cpp, “Optimizing Token Generation in llama.cpp’s CUDA Backend”
https://github.com/ggml-org/llama.cpp/discussions/17621
llama.cpp documentation, “Speculative Decoding”
https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md
llama.cpp issue, “Server: backend sampling gives +48% throughput at 32 slots”
https://github.com/ggml-org/llama.cpp/issues/27050
llama.cpp repository
https://github.com/ggml-org/llama.cpp