Last month I saw a someone loading a 70B parameter model on his Mac Studio and it was chugging along at something like 8 tokens a second. Not fast. But it ran. On his desk. No cooling tower, no separate GPU rig, no 1500 watt power supply screaming in the corner. I wondered and asked myself if this meant Apple had basically solved the GPU problem.
It hasn’t. Not even close, if you’re talking raw throughput. But the question stuck with me because it’s more interesting than a yes or no. There’s a real gap between “unified memory beats GPUs” and “unified memory does something GPUs structurally can’t,” and almost nobody explains the difference clearly. So I spent a weekend actually digging into bandwidth numbers, reading through llama.cpp benchmark threads, and trying to figure out where the line actually sits.
Short version, since I like to say where I land before I argue for it: unified memory will not catch discrete GPUs at throughput in general, not this decade. But for one specific job, running big language models locally without a rack of GPUs, it’s already good enough that the throughput gap stops mattering. That’s a narrower claim than people want to hear, and it’s the one that’s actually true.
Why GPUs Win the Numbers Game, Full Stop
Start with bandwidth, because that’s the number that actually decides most of this argument, not raw compute.
An H100 has HBM3 memory pushing around 3.35 TB/s. A 4090 on GDDR6X is around 1 TB/s. Apple’s best unified memory chip right now, the M3 Ultra, tops out somewhere near 800 GB/s. So even Apple’s flagship is sitting at roughly a quarter of what a real datacenter GPU pushes, and less than an Nvidia consumer card too.
Compute tells the same story, worse actually. Nvidia’s CUDA cores and tensor cores are purpose-built matrix multiplication machines with years of driver optimization behind them. Apple’s GPU cores are good, genuinely good for a mobile-lineage architecture stretched into desktop chips, but there just aren’t as many of them, and the software stack (Metal, MLX) hasn’t had the fifteen years of gaming and scientific computing investment CUDA has.
So if you’re asking “will an M4 Ultra ever out-throughput an H100 on a training run,” the honest answer is no. Not with the current architecture direction. Apple isn’t building for that job. It never has been.
What “Throughput” Even Means Here
I should slow down on this word because half the arguments I’ve seen online talk past each other over it. Throughput can mean tokens generated per second for one user. It can mean total tokens per second across a thousand concurrent requests on a server. It can mean raw FLOPS on a matrix multiply benchmark that has nothing to do with language models at all. People compare Apple to Nvidia using whichever definition makes their point, and that’s most of why this debate never resolves.
For this article I’m mostly talking about single-user tokens per second, because that’s the number that decides whether a person can actually use a model on their own hardware. It’s a narrower definition than “throughput” usually means in a datacenter context, and I want to be upfront about that instead of quietly switching definitions halfway through like a lot of these comparisons do.
Where Unified Memory Actually Punches Above Its Weight
What gets missed constantly, and it’s the whole reason this article exists.
Large language model inference, especially with a single user and low batch size, isn’t primarily a compute problem. It’s a memory bandwidth problem. Every token you generate requires the model to read its full set of weights from memory, and if your model is 70 billion parameters at even 4-bit quantization, that’s roughly 35–40GB you have to move through memory for every single token. Compute is almost idle. Bandwidth is the bottleneck.
This is why my friend’s Mac Studio, despite having far less raw compute than a 4090, can actually run a 70B model at all, while a single 4090 with 24GB of VRAM simply cannot, because the model doesn’t fit. You’d need at least two 4090s and some fairly annoying tensor-parallel setup to even attempt it, and even then you’re fighting PCIe transfer overhead between cards, which unified memory just doesn’t have because there’s no “between” to cross.
I remember seeing a thread on the llama.cpp GitHub from earlier this year where someone ran Llama 3.1 70B Q4 on an M2 Ultra and got something in the range of 8–10 tokens per second, while the same model split across two 3090s got maybe double that, but at a fraction of the power draw and with zero multi-card wrangling. I don’t have the exact thread bookmarked anymore, so take those numbers as ballpark, not gospel. But the shape of the result held up across a bunch of similar tests I looked at.
That’s the actual finding. Unified memory doesn’t win on speed. It wins on making a workload possible on one machine, quietly, without a server rack, and at speeds that are usable even if not blazing.
I actually tried to reproduce something close to this myself, borrowing the same friend’s Mac Studio for an afternoon, running a quantized Mistral model through LM Studio just to see the numbers with my own eyes instead of trusting screenshots from a thread. It took me way too long to figure out why my first few runs were slower than everything I’d read online, and it turned out I had Activity Monitor and two browser windows eating into the memory pool the model needed. Once I closed everything down the numbers lined up much closer to what people had reported. Small thing, but it’s the kind of detail that never makes it into the polished comparison articles, and it’s a good reminder that “unified” memory means the OS, apps, and model are all fighting over the same pool, which is its own quiet trade-off nobody mentions enough.
Quantization Is Doing More Work Than People Give It Credit For
None of this local-inference story works without quantization, and it’s worth explaining why briefly, since it changes the bandwidth math directly. A model at full 16-bit precision needs twice the memory movement per token compared to 8-bit, and four times compared to 4-bit. So dropping Llama 3.1 70B from FP16 down to a 4-bit quant doesn’t just save disk space, it roughly quarters the amount of data that has to move through memory for every token, which is the exact bottleneck unified memory is weakest on to begin with.
A couple of years ago, 4-bit quantization came with a real, noticeable quality hit. That’s gotten a lot better with newer quantization methods, and I’ve seen people argue a well-tuned 4-bit quant of a large model is closer in quality to the full-precision version than most people expect, though I’d stop short of saying it’s a solved problem, because you can still find specific tasks, long multi-step reasoning especially, where the quantized version noticeably loses the thread compared to full precision. So the “good enough” bar for unified memory keeps moving because quantization keeps quietly doing more of the heavy lifting, not because Apple’s chips got that much faster generation over generation.
Capacity Is the Real Weapon, Not Bandwidth
This is where I think most of the discourse gets it backwards. People keep comparing unified memory to GPU memory as if bandwidth is the whole argument. It isn’t. Capacity is doing most of the work here.
You can configure a Mac Studio with 512GB of unified memory. A single H100 tops out at 80GB. To match that capacity with Nvidia hardware you need multiple cards, and multiple cards means networking them, which means NVLink or InfiniBand, which means real money and real complexity. A 512GB Mac Studio, whatever you think of its price tag, is one box you plug in.
So the actual trade being made is: Apple gives up bandwidth and compute in exchange for capacity in a single unified pool, with no copy overhead between CPU and GPU because there’s only one pool of memory both sides read from directly. For a workload that’s bandwidth-bound but not desperately compute-bound, like single-user LLM inference, that trade works out better than the raw numbers suggest.
For a workload that needs both huge capacity and huge compute at the same time, like training a frontier model from scratch, it doesn’t work at all, because now you’re compute-starved on top of everything else.
Where It Falls Apart Completely: Training and Gaming
I want to be fair here and not oversell the good part, because that’s exactly the kind of imbalance this whole space is full of.
Training is nowhere near competitive on unified memory. Training needs sustained, enormous compute throughput across huge batches, not the memory-bound single-token generation pattern inference has. Every serious lab training frontier models is buying Nvidia H100s and now the Blackwell generation, not Mac Studios, and that isn’t going to change because Apple isn’t building silicon for that use case in the first place. Nobody at Anthropic or OpenAI is submitting a training run to a Mac Pro cluster and I’d bet money nobody ever will.
Gaming is the other place this argument just collapses. Games are not memory-bandwidth-bound the same way LLM token generation is. They need sustained parallel compute for rasterization, ray tracing, shader passes, and they need it at a stable 60 to 240 frames a second with almost zero tolerance for the kind of latency that’s fine for a chatbot response. Apple’s GPU cores, however clever the architecture, just don’t have the raw shader throughput of a 4090 or even a 4070, and the driver-level game optimization that Nvidia and AMD have spent two decades building doesn’t exist on Apple silicon in the same way. Metal is fine. It’s not CUDA plus DirectX plus a console-scale game industry building around your hardware for twenty years.
So if someone tells you unified memory is going to make discrete GPUs pointless for gaming, that’s just wrong, and honestly a little embarrassing to argue in 2026.
There’s a more honest version of the gaming argument, though, and it’s worth making instead of the exaggerated one. Apple has been slowly getting more games ported to Mac over the last couple of years, partly through Metal improvements and partly through translation layers that run Windows titles reasonably well. That’s a software and ecosystem story, not a hardware-throughput story. The unified memory architecture isn’t why more games run on a Mac now, better tooling and more developer attention are. It’s easy to conflate the two if you’re not paying close attention, and I’ll admit I did exactly that the first time I read about this, assuming Apple’s memory design was somehow behind the gaming improvements when it really wasn’t the main factor at all.
Training vs Inference: The Split That Actually Matters
If I had to compress this whole article into one sentence, it’s this: the GPU-vs-unified-memory question isn’t one question, it’s at least three, and they have three different answers.
Training: GPUs win, not close, won’t change soon. Batch inference at scale, the kind a company runs to serve millions of API requests: GPUs win, because throughput per dollar per watt at scale is what matters, and Nvidia’s compute density plus mature software stack (CUDA, TensorRT, vLLM) wins that fight decisively. Single-user local inference on a big
Model: this is the one place unified memory is already good enough, and in some capacity-bound cases, the only practical option on consumer hardware at all.
That third category is small compared to the first two in terms of total compute spent globally. But it’s the category that individual people, hobbyists, small startups, privacy-conscious folks who don’t want their prompts hitting someone else’s server, actually live in. And it’s growing.
What This Does to Pricing, If the Gap Keeps Closing
This is the part I think gets underrated. If Apple (or anyone else building unified memory architectures, and there are a few now looking at similar designs) keeps closing the usable gap for local LLM inference, it changes the pricing conversation in a specific way, not a dramatic one.
Right now, if you want to run a big open model locally, your two real options are: buy multiple high VRAM GPUs and deal with the complexity, or buy a Mac with a lot of unified memory and accept slower tokens per second in exchange for simplicity and one box. As models keep growing but quantization techniques keep getting better (4-bit and even lower-bit quantization losing less quality than people expected a couple years ago), the “just enough tokens per second to be usable” bar keeps getting easier to clear on unified memory.
That doesn’t threaten Nvidia’s core business, which is training and datacenter inference at scale, where compute density is still the whole game. What it does threaten is the narrower market of people buying two or three consumer GPUs specifically to run big models at home. That’s a real but small dent, not an earthquake. I don’t think Jensen Huang is losing sleep over it, and I’d be overselling this if I said otherwise.
There’s also a secondhand market effect worth mentioning, one I hadn’t thought about until I was digging through this. As unified-memory Macs become a viable local inference option, some of the demand that used to go toward used 3090s and 4090s for hobbyist LLM rigs might soften slightly, which could actually help GPU prices for gamers, who’ve been dealing with inflated used-market prices for years partly because of crypto mining and partly because of exactly this local-AI hobbyist demand. That’s speculative on my part, I haven’t seen hard data on it, so treat it as a guess rather than a claim.
And then there’s the API pricing side, which is a completely different conversation from consumer hardware but people mix the two up constantly. Companies like Anthropic and OpenAI aren’t setting their per-token API prices based on what a Mac Studio can do. Those prices come from datacenter-scale batch inference economics, where GPU utilization across thousands of concurrent requests is the whole game, and unified memory architectures at consumer scale don’t touch that market at all right now. If unified memory ever does show up at datacenter scale in a serious way, and Nvidia’s own Grace Hopper direction suggests even they think there’s something there, it would be competing with H100 and Blackwell clusters, not with a Mac Studio on someone’s desk. So don’t expect your ChatGPT or Claude subscription price to move because Apple shipped more memory bandwidth. Those are just unrelated markets wearing similar-sounding names.
Apple Specifically, and Why It’s the Best Example of This
I keep coming back to Apple in this article because it genuinely has the best unified memory architecture on the market right now, not because of brand loyalty, I don’t even own a Mac Studio myself, I’ve just used a friend’s for testing.
The M-series chips put CPU, GPU, and memory controller all on one piece of silicon with a shared memory pool, so there’s no PCIe bus between compute and memory the way there is in a traditional CPU-plus-discrete-GPU setup. That single design decision is what makes the huge memory configurations (192GB, 512GB) actually usable for GPU-bound work at all, because the GPU cores can address that entire pool directly instead of copying chunks across a bus.
Nvidia’s Grace Hopper superchip is doing something conceptually similar at the datacenter level, unifying CPU and GPU memory, and it’s worth watching because it suggests even Nvidia sees value in this direction for certain workloads. But that’s a very different price and scale tier than a consumer Mac, so it doesn’t really compete in the same conversation people are having about home LLM setups.
There’s an unresolved detail worth mentioning here, something that was still being argued about in forum threads as of a few weeks ago. People have noticed that memory bandwidth utilization on the newest M-series chips doesn’t scale linearly with the advertised peak number once you’re running an actual LLM workload instead of a synthetic benchmark, sometimes landing noticeably below the spec sheet figure depending on the exact model and context length. Nobody’s fully explained why yet, whether it’s a driver-level scheduling issue, thermal throttling under sustained load, or just an artifact of how llama.cpp’s Metal backend handles certain operations. Apple hasn’t commented on it publicly as far as I’ve seen. So the real-world number is probably a bit worse than the spec sheet in some cases, and I don’t think anyone outside Apple’s own silicon team actually knows the full reason why yet.
I bring up AMD here too because it’s easy to forget they exist in this conversation. AMD’s Strix Halo chips are chasing something similar to Apple’s approach, a unified memory pool shared between CPU and GPU on one package, aimed specifically at making large-model inference possible on a single machine without discrete GPU memory limits. It’s early days for that lineup compared to Apple’s several generations of M-series chips, but it tells you this isn’t an Apple-only idea. It’s a broader recognition that for this one specific workload, memory-bound single-user inference, the traditional discrete-GPU-with-limited-VRAM model has a real weakness that unified architectures are built to exploit.
So, Will It Ever Catch Up?
Not at throughput, no. And I don’t think that’s actually the right question anymore, even though it’s the one everyone keeps asking.
The better question is whether unified memory closes the usable gap for enough workloads that the throughput difference stops mattering to the person using it. For single-user local LLM inference, that’s already mostly true today, with the caveat that “mostly true” still means single digit to low double digit tokens per second on the biggest models, which some people find genuinely too slow and others find perfectly fine for a personal assistant that doesn’t need to respond instantly.
For gaming, batch inference, and training, the gap isn’t closing in any meaningful way, and I don’t see an architectural path where it would, short of Apple deciding to build something that looks a lot more like a discrete GPU with its own dedicated high-bandwidth memory, at which point it stops really being “unified memory” in the sense people mean when they ask this question.
If you’re deciding what to actually buy based on this: buy a GPU if you’re gaming, training, or serving inference at any real scale. Consider a high-memory Mac only if your specific use case is running big open models locally, alone, and you value simplicity and quiet over squeezing out every extra token per second. That’s a narrower recommendation than most of the hype pieces around this topic give you, but it’s the honest one.