The Mac did not get a new GPU.
Apple did not unlock a hidden performance mode.
Qwen3.8–27B did not suddenly lose half its parameters.
Yet on the same 48GB M5 Pro, a new local inference engine called Splash is reporting around 74 tokens per second on short prompts with Qwen3.8–27B and about 54 tokens per second at 32K context.
With four requests running at once, combined decode throughput reaches roughly 170 tokens per second.
The hardware is the same. The model is the same. What changed is the software sitting between them.
That is why I think Splash is one of the more important local-AI releases this month.
Not because every Mac has suddenly become twice as fast. Not because llama.cpp, MLX, Ollama or every other local runtime is obsolete.
The interesting part is that Splash is built around a completely different assumption:
What if an inference engine stopped trying to support every model and instead optimized itself around one specific model?
That sounds like a small implementation detail. It may be the beginning of a much bigger shift in local AI.
Access without medium partner: LM Studio Splash M5 Pro Performance
First, the number everyone is going to repeat
Inco AI, the company behind Splash, tested Qwen3.8–27B on an M5 Pro with a 16-core GPU and 48GB of unified memory.
Its reported short-prompt decode speed was 74 tok/s. The next-fastest engine in Inco’s comparison was around 38 tok/s.
At 32K context, Splash reportedly held around 54 tok/s, while the next-fastest comparison was roughly 28 tok/s.
That is close to a 2× lead.
Then the comparison becomes more interesting when several requests run together. With four concurrent short-prompt requests, Splash reached about 170 tok/s aggregate throughput on Qwen3.8–27B. The next-fastest engine in Inco’s comparison was around 43 tok/s.
That is roughly 3.9×.
Those numbers come from Inco’s own launch benchmarks, not from an independent lab. That caveat matters. The tests used specific coding prompts, recommended settings for each engine, reasoning enabled, and Inco’s own Splash package.
So I would not turn “74 tok/s” into a universal promise for every 48GB Mac.
But even with that caveat, the architecture behind the result is worth paying attention to.
LM Studio did not invent Splash
There is an important distinction here.
Splash is built by Inco AI.
LM Studio integrated it as a first-class runtime in LM Studio Bionic 1.1.5 and then added Splash support to LM Studio 0.4.25 on September 19.
That is why the story feels like an LM Studio update from the user side.
Open LM Studio. Go to the runtime settings. Download Splash. Load the supported Qwen package.
The experience looks like another runtime option.
Underneath that simple button is a very different inference philosophy.
Most local runtimes are designed around flexibility. They want to support many architectures, many quantizations, many context sizes, many machines and thousands of model files.
Splash does almost the opposite.
It currently supports only a tiny set of models.
And that limitation is the feature.
General-purpose inference has always paid a flexibility tax
Think about what llama.cpp has to do.
It supports an enormous number of model families. It supports CPUs. NVIDIA GPUs. AMD GPUs. Apple Silicon. Different quantization formats, context lengths, architectures and offload strategies.
That flexibility is one of the reasons llama.cpp became such an important piece of local-AI infrastructure.
The same broad idea applies to Ollama and other general-purpose runtimes.
They are useful because the runtime does not need to be rebuilt every time someone downloads a different model.
But flexibility has a cost.
The engine needs generic code paths. Its kernels have to handle more than one exact tensor shape. Its memory planner has to work across many different models. Its scheduler cannot assume every request looks the same. Its speculative decoding system may need to support different draft strategies.
Splash decides that this is wasted opportunity.
For each supported model, Inco specializes almost everything: the kernels, the draft model, the memory plan, and the performance baselines.
The shared parts are things such as the runtime, scheduler, cache and API.
Everything else can be tuned around the exact model.
That is why Splash can be faster.
It knows exactly what it is serving before it starts.
Qwen3.8–27B is not being loaded like an ordinary model
The Splash package for Qwen3.8–27B is about 17.4GB.
That package includes the 4-bit target model, a dedicated DFlash 2 speculative draft, the vision encoder and tokenizer.
It is not a normal Transformers checkpoint.
It is not an MLX checkpoint.
It cannot simply be loaded by another random inference engine.
It is packaged specifically for Splash.
That is another clue about what is happening.
Instead of asking, “How can one engine support everything?” Splash asks, “What is the fastest package we can build for this model on this hardware?”
Those are two very different engineering goals.
The biggest trick is speculative decoding that is not optional
Normal autoregressive generation is awkwardly sequential.
The model predicts one token. Then another. Then another.
Even with a very fast GPU, that token-by-token dependency limits how much parallel work the hardware can do.
Speculative decoding tries to get around that.
A smaller draft model predicts several likely future tokens. The main model verifies those candidates together. If several guesses are correct, the target model effectively moves forward by multiple tokens during one verification step.
Splash does not treat speculative decoding as a bonus setting.
It is the normal decode path.
Every supported model comes with its own DFlash 2 draft model trained specifically for that target.
That matters because speculative decoding is only useful when the draft predicts well enough to save more time than it costs.
A generic draft can help. A draft trained around one target model can potentially help much more.
And Inco does not stop at attaching the draft.
It also builds the execution path around it. Drafting, verification, token acceptance and state updates are designed to work together instead of being several loosely connected operations.
This is where software optimization starts to feel like hardware optimization.
The kernels know the exact shapes they are going to see
This is probably the least glamorous part of the story.
It may also be the most important.
GPU inference depends heavily on kernels: small programs that perform matrix multiplications, attention operations, expert routing and other model computations.
A general-purpose engine needs kernels that work across many models and tensor shapes.
Splash generates and tunes kernels for the exact dimensions used by the model.
Inco says its internal kernel agents generate shape-specialized Metal kernels for different parts of the workload.
Prefill gets kernels designed for large batches. Decode gets kernels designed for small verification blocks. Hybrid recurrent components get specialized paths. Mixture-of-experts routing gets its own kernels.
The important phrase is exact shapes.
When the runtime knows the dimensions ahead of time, it has fewer compromises to make.
The result is closer to software written for one machine and one workload than a framework trying to support an entire model ecosystem.
That approach obviously does not scale cheaply across ten thousand models.
For the models that receive the optimization work, it can be extremely effective.
The 48GB recommendation is not because the model itself needs 48GB
This part is easy to misunderstand.
Splash requires at least 36GB of unified memory, and Inco recommends 48GB or more.
But the Qwen3.8 Splash package itself is only around 17.4GB.
So why recommend 48GB?
Because running the model is only part of the memory budget.
Splash needs room for the target weights, the draft model, KV cache, recurrent state, batching, application overhead and the rest of macOS.
A real coding-agent session also means the user probably has an editor, browser, terminal, build tools and other applications open.
Inco says the Qwen3.8 target uses about 15GiB of weights, while its draft adds around 1.2GiB before context cache is counted.
The extra memory is headroom.
This is exactly where Apple’s unified-memory design helps.
The GPU and CPU operate from one large memory pool instead of being separated into normal system RAM and fixed dedicated VRAM.
That does not make Apple Silicon magically faster than discrete GPUs.
It makes memory allocation unusually flexible.
Splash appears to take advantage of that flexibility aggressively.
The 32K result matters more than the 74 tok/s screenshot
Short-prompt token speed is easy to market.
Agents do not stay at short prompts for very long.
A coding agent reads files. Then it reads logs. Then it calls tools. Then it adds test results. Then it edits code. Then another tool result appears.
The context grows.
A benchmark that starts at 74 tok/s and collapses at 16K or 32K is less useful than it initially looks.
That is why the 54 tok/s at 32K context number is interesting.
Inco reports that Splash maintains a significant lead as context grows.
The engine also uses a paged KV cache with prefix reuse. If a new request shares a prefix with earlier work, cached state can be reused instead of recomputing the same history.
For hybrid models with recurrent layers, Splash also snapshots relevant recurrent state so prefix caching can cover more than ordinary attention.
This is the kind of work that makes an agent feel fast after thirty minutes, not just during a five-second demo.
Cached time-to-first-token may be the more important number
There is another benchmark that deserves more attention than raw decode.
On a cached 32K replay, Splash reports around 282 milliseconds time to first token for Qwen3.8–27B.
Inco says that was around 7.3× faster than the next-fastest engine it measured for that scenario.
This matters because an agent repeatedly revisits the same context.
It does not always need to reread the entire repository from zero.
If the runtime can reuse prefix state effectively, later turns can start much faster.
Raw tokens per second measures how quickly the model talks once generation begins.
Time to first token measures how long the user stares at nothing.
Both matter.
For interactive agents, the second one can dominate how fast the system feels.
The cold-prefill result is less magical
This is where Splash becomes more honest.
Cold-reading a large context is still expensive.
On Qwen3.8–27B at a 32K prompt, Splash reportedly processes around 363 input tokens per second.
That was faster than the other engines in Inco’s comparison, but not by the same dramatic margin seen in cached replay or concurrency.
Inco reports roughly 96 seconds to the first token for a cold 32K prompt in that benchmark.
That is a long time.
So no, Splash does not make local inference instantaneous.
It does not repeal the cost of reading a large repository for the first time.
The improvement becomes much more valuable once the context is loaded and reused.
That distinction is exactly the kind of detail benchmark headlines usually leave out.
Four subagents may be where Splash makes the strongest case
The local-AI market is changing.
A year ago, most people cared about one chat response.
Now agents are spawning agents.
A coding system may ask one subagent to inspect tests, another to read documentation, another to investigate an error and another to review the proposed fix.
That means concurrency matters.
One request at 74 tok/s is useful.
Four requests sharing one GPU is a different problem.
Splash reports around 170 tok/s aggregate decode throughput for four concurrent short-prompt Qwen3.8 requests on the same 48GB M5 Pro.
That is where its lead widened to around 3.9× over the next-fastest engine Inco measured.
Why?
Because Splash knows exactly how much memory the model requires. It knows the state cost per request. It knows how the draft behaves. It can batch requests and allocate memory much more aggressively because there are fewer unknowns.
Inco also reports that on 16 concurrent 32K requests, Splash accepted and completed all 16 on that 48GB machine, while a general-purpose memory policy accepted nine on its first pass.
That is not simply a faster-matrix-multiply story.
It is a scheduling story.
Local AI may be entering the runtime era
The first local-AI hardware discussion was simple.
What GPU do you have?
Then people learned to ask how much VRAM it had.
Then unified-memory systems changed the conversation again.
Now we are reaching another stage.
What inference engine are you using?
That question can change the result enough that two people with the same Mac and the same model may report very different experiences.
One runtime may use generic kernels. Another may use a trained speculative draft. One may repeatedly process the same prefix. Another may reuse it. One may reserve memory conservatively. Another may know the exact model layout and safely use almost all available memory.
The hardware specification did not change.
The useful performance did.
That is a strange place for PC benchmarking to be.
It is also probably where local AI was always going.
This should make hardware buyers a little uncomfortable
People spend thousands of dollars chasing inference speed.
Buy a Max chip instead of a Pro. Buy more GPU cores. Move from 48GB to 64GB. Build an RTX workstation.
Then a software update appears and changes performance by a larger percentage than some hardware upgrades.
We just saw a similar argument on NVIDIA with llama.cpp optimizations.
Splash makes it even more obvious because the specialization is so aggressive.
This does not mean hardware stops mattering.
Obviously it matters.
A slow GPU does not become a fast GPU through positive thinking.
Memory bandwidth still matters. GPU compute still matters. Memory capacity still matters.
But hardware determines the ceiling only after software decides how much of that ceiling gets used.
Local inference software is still young enough that there is a lot of unused performance hiding in machines people already own.
There is a giant catch: Splash supports almost nothing
This is the criticism that needs to sit directly beside the impressive benchmark.
Splash currently supports a small set of models.
At launch, the main supported packages are Qwen3.8–27B and Qwen3.6–35B-A3B.
That is it.
If you want Llama, no.
If you want Gemma, no.
If you want a random Hugging Face fine-tune, not unless someone builds a Splash package for it.
If you want to point the engine at an ordinary MLX checkpoint, no.
This is the price of specialization.
A universal engine is useful because the model ecosystem moves absurdly fast.
Splash gets its performance by refusing to be universal.
That may be completely fine.
It just means the comparison with llama.cpp is not really “which engine is better?”
The comparison is:
Do you want maximum flexibility, or maximum optimization for a small set of models?
Those are different products.
The benchmark comparison is also not perfectly controlled
Inco says every engine was tested using its recommended settings.
That is reasonable from an end-user perspective.
It also means the comparison measures entire configured systems, not one isolated optimization.
Splash has its specialized draft. Other engines may use different quantization. Some support batching differently. Some have different cache strategies.
Inco explicitly says the numbers show the combined effect of specialization rather than the contribution from one technique.
That is the correct interpretation.
And again, these are vendor benchmarks.
I want to see independent testers reproduce the numbers across several M3, M4 and M5 configurations.
I want long-running coding sessions.
I want power consumption.
I want memory pressure with Chrome, Xcode and Docker open.
I want to see what happens when the context reaches 64K or 128K.
I want speed measurements after thermal equilibrium, not just short runs.
Until then, 74 tok/s should be treated as Inco’s measured result, not a universal fact about every 48GB Mac.
M3 or newer leaves a lot of Macs outside the party
Splash also has fairly narrow hardware requirements.
It currently requires Apple M3 or newer, macOS 26.4 or later, and at least 36GB unified memory. Inco recommends 48GB or more.
That means many otherwise excellent Apple Silicon machines cannot use the current Splash path.
A 32GB Mac does not meet the published minimum.
An M2 Max with a large amount of memory does not qualify because the minimum architecture is M3.
That matters because the local-AI community still contains a lot of M1 Max, M2 Max and older Mac Studio hardware precisely because those machines offered unusually large unified-memory pools.
The specialized engine improves performance.
It also narrows compatibility.
Again, specialization always sends an invoice.
Qwen3.6 may actually have the crazier benchmark
Qwen3.8–27B gets the cleaner headline.
The Qwen3.6–35B-A3B result is arguably more absurd.
Inco reports 210 tok/s on short prompts on the same 48GB M5 Pro.
At 32K context, the number remains around 143 tok/s.
With four concurrent short requests, aggregate throughput reaches around 357 tok/s.
There is an architectural reason.
Qwen3.6–35B-A3B is a Mixture-of-Experts model with only a fraction of its total parameters active during each token.
That makes the comparison very different from a dense 27B model.
Still, seeing a 35B-class model stream at more than 200 tokens per second from a laptop-class Apple Silicon system tells you how strange local inference optimization is becoming.
Parameter count alone is no longer enough to predict performance.
Neither is hardware alone.
Architecture plus runtime plus quantization plus memory layout now matter enormously.
This is exactly what Apple Silicon needed
Apple’s local-AI advantage has always been slightly frustrating.
The hardware offers large unified-memory configurations and very good power efficiency.
The software ecosystem around local inference has improved dramatically through MLX, llama.cpp and tools like LM Studio.
NVIDIA still dominates the broad AI tooling conversation because CUDA is everywhere.
Splash does not change that.
What it does show is that Apple Silicon can benefit from software designed specifically around Metal and unified memory, rather than treating the Mac as another target to support after CUDA.
That is important.
The fastest way to make Apple hardware more competitive in local AI may not be waiting for the next M-series chip.
It may be writing inference systems that assume Apple’s memory architecture from the beginning.
Splash is one version of that idea.
The real question is whether specialization scales
This is the part I keep coming back to.
Splash looks great when there are two supported models.
What happens when users want twenty?
Each model needs a specialized package. Each model needs tuned kernels. Each model needs a draft. Each model needs performance testing. Each hardware family may need presets.
That is a lot of engineering.
Inco says its internal kernel agents automate part of that work.
If that automation is genuinely effective, the model-specific approach could scale farther than it sounds.
Instead of humans hand-tuning every kernel, AI systems could generate and benchmark specialized kernels whenever a new important model appears.
That creates a weird recursive loop:
AI helps write the software that makes AI run faster.
If that process becomes cheap enough, a universal engine may not need to be the fastest engine.
It may simply be the compatibility layer.
The popular models get specialized runtimes.
Everything else falls back to the general-purpose engine.
That feels plausible.
The future may look like game optimization profiles
PC gamers already understand this idea.
A graphics driver can contain game-specific optimizations.
A console game can be tuned around one exact hardware configuration.
Performance comes not only from the GPU but from knowing the workload.
Local AI may move in the same direction.
Install a model.
The runtime recognizes it.
It downloads a model-specific kernel package.
It selects a trained speculative draft.
It chooses a memory plan for your exact chip and RAM configuration.
Then the application exposes all of that as one button.
The user never thinks about kernels.
They just notice the same Mac feels faster than it did yesterday.
That is basically what Splash is demonstrating.
Your Mac didn’t get faster. Our definition of the Mac did.
A computer is not only its silicon.
It is the entire execution stack sitting on top of that silicon.
That sounds obvious until a software update produces a performance jump large enough to compete with the improvement people expect from buying another machine.
Splash does not prove that model-specific inference engines will replace llama.cpp.
It does not prove every Qwen user should immediately abandon Ollama.
It does not prove a 48GB Mac is suddenly equivalent to a giant NVIDIA workstation.
What it proves is more interesting.
Local inference is still inefficient enough that changing the runtime can radically change the machine you think you own.
On Inco’s benchmark, the same 48GB M5 Pro goes from ordinary Qwen3.8 performance to roughly 74 tok/s, stays near 54 tok/s at 32K context, and reaches about 170 tok/s aggregate throughput across four concurrent requests.
No new GPU.
No new RAM.
No new Mac.
Just a runtime that stopped trying to be everything for everyone.
That trade is the story.
The next local-AI upgrade may not be another $3,000 computer.
It may be a Download Runtime button.