The first number that sells NVIDIA DGX Spark is 128GB. That is the number local LLM people notice because 128GB means models that simply refuse to fit on a normal gaming GPU can finally sit on one small desk computer.
Then you load Llama 3.1 70B in FP8 and something weird happens. The model fits. The prompt can be processed at around 803 tokens per second. But when the model starts writing its answer, LMSYS measured only 2.7 tokens per second on a single DGX Spark. A 70B model is sitting inside 128GB of unified memory, the machine has Blackwell hardware, NVIDIA advertises up to 1 PFLOP of FP4 AI compute, and yet the reply comes out slowly enough that you can watch every few words arrive. That sounds broken. It isn’t. The box is running into a different limit. (LMSYS)
And this is exactly why I think DGX Spark is one of the best machines for explaining what local AI hardware numbers actually mean. More memory does not automatically mean more speed. A huge compute number does not automatically mean faster chat. For large dense models, one less glamorous number can matter much more: memory bandwidth.
128GB answers one question. It does not answer the next one
DGX Spark has 128GB of coherent LPDDR5X unified system memory. NVIDIA currently lists memory bandwidth at 273GB/s, a 20 core Arm CPU, Blackwell GPU, 4TB NVMe storage on the main configuration, and up to 1 PFLOP of FP4 tensor performance with sparsity. NVIDIA also says the machine can run inference on models up to 200B parameters and fine tune models up to 70B. (NVIDIA)
Those are unusual specs for a box measuring roughly 150mm by 150mm by 50.5mm.
The 128GB number is genuinely useful. An RTX 5090 has 32GB of GDDR7. That memory is much faster, but 32GB is still 32GB. A 70B model in FP8 needs roughly 70GB just for weights before you count the KV cache, runtime allocations, and other overhead. It cannot live fully inside a single 5090 at that precision. Spark can hold it without splitting the model across several consumer GPUs or pushing a large part of it into ordinary system RAM.
That sounds like a win. It is a win, for capacity.
Speed is another job.
Think about what happens while an LLM writes a reply. The prompt has already been read. Now the model predicts one token, then the next one, then the next. For a dense model, almost the whole set of active weights has to be read again and again as new tokens are produced. If the model has around 70GB of weights and the memory system can move 273GB each second in perfect conditions, you can already see the problem with simple division.
273 divided by 70 is about 3.9.
That is an ideal ceiling before real software overhead, memory access inefficiency, routing, cache traffic, and everything else gets involved. LMSYS measured 2.7 tokens per second for Llama 3.1 70B FP8. Suddenly the result does not look mysterious at all. The machine is actually getting a decent fraction of what the memory system can physically feed to a dense model of that size. (LMSYS)
Nothing has crashed. Nothing is badly configured. Physics is doing the limiting.
The 1 PFLOP headline is not the number you feel during chat
This part trips people up because AI hardware marketing has trained us to look for compute numbers first.
NVIDIA says DGX Spark can reach up to 1 PFLOP at FP4 precision using sparsity. That tells you the tensor hardware can do a massive amount of low precision math under the right workload. It matters for prompt processing, training related work, batched workloads, and parts of inference that can keep those tensor units busy. (NVIDIA DGX Spark hardware guide)
But chat has two different phases.
First comes prefill. The model reads your prompt. This can be heavily parallel, so compute matters a lot. That is why LMSYS saw the same 70B model process the prompt at around 803 tokens per second.
Then comes decode. The model writes the answer token by token. For a large dense model, decode often turns into a memory traffic problem. The GPU has plenty of math capability sitting there, but it cannot calculate on weights that have not arrived from memory yet.
So the strange result makes sense: 803 tokens per second in one phase, 2.7 in the next.
If you only read the 1 PFLOP number, you might expect the whole experience to be very fast. If you read 273GB/s beside it, the dense 70B result becomes much easier to predict.
This is the spec I would look at first when somebody shows me another 128GB AI mini PC.
Then a 120B model comes along and runs far faster
Here is where the story becomes fun.
The llama.cpp maintainers have published DGX Spark results for OpenAI’s gpt oss 120B model. The model has about 117B total parameters, yet the benchmark shows around 58.7 tokens per second for a single stream token generation test in the tested MXFP4 configuration. That is more than twenty times the 2.7 tokens per second LMSYS measured for dense Llama 3.1 70B FP8. (llama.cpp benchmark)
A 120B model beating a 70B model by that much sounds completely backwards until you look at the architecture.
OpenAI says gpt oss 120B has 116.8B total parameters but only about 5.1B active parameters for each token. It is a Mixture of Experts model. Instead of sending every token through the whole model, it routes work through a small set of experts. The full model still needs memory capacity, but far less of it is active for each token. (OpenAI)
That is basically DGX Spark’s favorite kind of model.
The machine has a large memory pool, so it can hold a big collection of experts. The model activates only a small part at a time, so it does not need to stream 70GB of dense weights for every generated token. The bandwidth pressure falls sharply.
This is why parameter count by itself has become a bad shortcut for predicting local LLM speed.
A dense 70B model can be painfully slow. A 117B MoE model can feel fast. A 200B sparse model may make more sense on Spark than a much smaller dense model. You have to know how much of the model actually works on every token and what quantization format the runtime is using.
That is also why I would be careful with articles that rank local AI computers using only the largest parameter count they can load. Fitting a model is step one. Getting a useful response speed is step two.
Quantization helps, but it cannot make bandwidth disappear
There is an obvious fix for the 70B problem: make the model smaller.
If you move from FP8 to roughly 4 bit weights, a 70B model may shrink from around 70GB toward the 35GB to 40GB area depending on format and overhead. That means less data has to cross memory for each token. The speed should go up.
And it does.
A recent reproducible test of Llama 3.3 70B in NVIDIA’s NVFP4 format on GB10 hardware measured about 5.4 tokens per second for a single stream. The researcher estimated a realistic ceiling around 5.5 tokens per second for that setup because roughly 39GB of weight reads dominate the work. (DGX Spark Research)
That is roughly twice the old FP8 result. Better. Still not what I would call fast chat.
And there is another cost. More aggressive quantization can reduce model quality. The amount depends on the model, format, task, and quantizer. Sometimes the loss is tiny. Sometimes coding, math, long reasoning, or uncommon knowledge takes a hit. You cannot look at a 4 bit file and assume it behaves exactly like the higher precision model.
So quantization gives Spark more room and more speed, but the basic relationship stays there. Large dense models move lots of weights. Spark has 273GB/s to move them.
A software update can improve kernels, scheduling, speculative decoding, cache use, and overhead. Those changes matter. LMSYS found speculative decoding could improve end to end inference on tested models. But software cannot turn 273GB/s LPDDR5X into 1.8TB/s GDDR7 by being clever. (LMSYS)
The RTX 5090 comparison makes Spark look slow and smart at the same time
An RTX 5090 is a useful comparison because NVIDIA lists it with 32GB GDDR7 and 1,792GB/s of memory bandwidth. That is more than six times Spark’s 273GB/s. (NVIDIA)
For models that fit inside 32GB, the 5090 can be a monster. Its memory can feed weights far faster, and its GPU is built for much higher throughput. If your daily model is 8B, 14B, or a compressed 30B class model, buying DGX Spark just because it has 128GB can make little sense if speed is your main goal.
Then try loading a 70GB FP8 model onto the 5090.
You can’t keep the whole thing in 32GB VRAM.
Now you need heavier quantization, CPU offload, multiple GPUs, or some mix of those. Each choice adds a compromise. Spark’s 128GB starts looking useful again because the model can sit in one coherent memory pool without PCIe shuffling between a CPU and several cards.
This is why saying “RTX 5090 is faster than DGX Spark” is both true and incomplete.
Faster at what?
Spark is not really a tiny replacement for a 5090 gaming PC. It is a different type of machine. NVIDIA built it around local AI capacity, CUDA development, large models, agent experiments, and a path toward bigger NVIDIA systems. Its strength is that 128GB pool sitting beside Blackwell compute and NVIDIA’s software stack.
Its weakness is that the pool is much slower than high end discrete GPU memory.
Both things are true at the same time.
The 200B claim needs the same reality check
NVIDIA says DGX Spark can run inference with models up to 200B parameters. I do not think that claim is fake. The 128GB memory pool plus low precision formats can make very large models fit. NVIDIA also supports connecting two Spark systems for models up to 405B. (NVIDIA)
But “up to 200B” tells me almost nothing about how the model will feel.
Is it dense or MoE? How many parameters are active per token? Is it FP8, FP4, or another quant? How large is the KV cache? What context are you using? Is there one user or thirty? Which inference engine? Is the task interactive chat or overnight batch work?
Those questions can matter more than the 200B number.
A dense 100B plus model may fit after compression and still write answers at a pace that becomes annoying. A larger MoE model with low active parameter count can be much faster. This is the strange part of buying local AI hardware in 2026: the model architecture now affects your hardware experience so much that “maximum model size” is becoming close to a marketing shorthand.
You need the second line of the spec sheet.
2.7 tokens per second is bad for chat, but Spark is not useless
I would not want to use a dense 70B model at 2.7 tokens per second as my everyday assistant. A long answer becomes a waiting exercise. If I am jumping between code, browser tabs, and prompts all day, I want the model to keep up with me.
But that does not make DGX Spark a failed product.
It means dense single stream chat is one of its weaker uses when the model gets large.
Spark makes much more sense for MoE models, local experiments that need more than 32GB of model memory, large context work, fine tuning, CUDA development, privacy sensitive workloads, and multi user or batched jobs where total throughput matters more than one stream. The llama.cpp gpt oss 120B benchmark is a good example of how different the machine feels when the model architecture fits the hardware. (llama.cpp)
Concurrency changes the story too. A single dense stream may crawl because it is waiting on memory. Several requests can keep more of the machine busy and increase total throughput even though each individual user does not suddenly get hundreds of tokens per second. That matters if Spark is acting as a small office server or feeding several agents at once.
And this is probably the part I underestimated when I first looked at the 2.7 number. I saw slow chat and thought the hardware looked badly matched. It is badly matched for that exact workload. Give it a sparse MoE model or several parallel jobs and the picture changes very fast.
The mistake is expecting one benchmark to describe the whole box.
What I would run on DGX Spark
If I owned a Spark today, I would not start by hunting for the largest dense model that fits.
I would start with MoE models. gpt oss 120B is an obvious example because OpenAI says only 5.1B parameters are active per token and llama.cpp has already shown around 58.7 tokens per second generation on Spark in one published setup. I would also look closely at current Qwen MoE models and other sparse models that keep active weight traffic low. (OpenAI)
For dense models, I would probably stay much smaller unless I had a task where model quality mattered more than response speed. A dense 70B at 4 bit around five tokens per second can still be useful for batch document work, offline analysis, slow research jobs, or a second pass where I do not care about watching the answer arrive.
That is less sexy than saying “run 200B AI on your desk.” It is also much closer to how I would spend $4,000 plus of my own money.
The lesson is simple.
128GB tells you what can fit.
273GB/s tells you a lot about how a big dense model will feel once it starts talking.
And the difference between those two numbers is basically the entire DGX Spark story.