So you open up Claude or ChatGPT one day and there's a new number next to the name. Opus 4.5 became 4.6, then 4.8. GPT-4 became GPT-5. And somehow the thing just... knows more, codes better, doesn't hallucinate as much. What actually happened in the lab between those two version numbers?
I used to think it was just "they added more data, made the model bigger, done." That's what most people assume, honestly. And it's not wrong exactly, but it's only maybe 30% of the real picture in 2026. The other 70% is a mix of things that don't get talked about as much: how the data is filtered, what happens after pretraining, and increasingly, how much the model is allowed to "think" before it answers you.
Let me walk you through it the way I understand it, using real models as examples where I can.
Bigger Models Used to Be the Whole Story
Back in 2020, a paper from OpenAI (Kaplan et al.) showed something that shaped the next four years of AI: if you make a model bigger, and feed it more data, and give it more compute, its performance improves in a predictable way. Not randomly, predictably, like a curve you could actually plot. This is what people call scaling laws.
The idea was simple to state, if hard to execute: throw more parameters at the problem and the loss goes down. GPT-2 had 1.5 billion parameters. GPT-3 jumped to 175 billion. That's not a small jump, that's more than 100x. And GPT-3 was noticeably better at basically everything — writing, reasoning, translation, you name it.
But here's where it gets interesting. In 2022, DeepMind published something called the Chinchilla paper, and it kind of broke the assumption everyone was working off. They found that most big models at the time were actually undertrained labs were making models huge but not feeding them enough data relative to their size. A smaller model trained on way more tokens could beat a bigger model trained on fewer tokens, for the same compute budget.Chinchilla itself was 70B parameters, way smaller than GPT-3's 175B, and it still won on most benchmarks.
This changed how everyone thought about scaling. It's not "bigger is always better." It's "bigger AND fed enough data is better," and the two have to grow together, roughly proportionally. If you only crank up parameters without matching data, you're leaving performance on the table. If you only add data without more parameters, the model can't hold onto it all.
Then We Hit a Wall, the Data Wall
Here's the part that's kind of a mess right now, and most explainer articles gloss over it. By around 2024–2025, labs started running into a real problem: the internet only has so much high-quality human-written text. Estimates for how much "usable" text exists vary, but everyone agrees it's finite, and the big labs have basically scraped most of what's out there already, books, papers, code repos, forums, the works.
So what do you do when you can't just get more real data? You've got a few options, and honestly none of them are as clean as "add more data."
One option is synthetic data, you use an existing strong model to generate new training examples, then train the next model on those. This sounds a bit circular (and it kind of is), but done carefully, especially for math and code where you can automatically check if an answer is correct, it works surprisingly well. DeepSeek did a lot of this with their R1 model, using it to generate reasoning traces that then trained smaller, cheaper models.
Another option is being way more picky about data quality instead of just quantity. Instead of throwing every scraped webpage into the pretraining mix, labs now filter aggressively removing duplicate content, low-quality spam, and stuff that doesn't actually teach the model anything new. A well-curated 5 trillion tokens can beat a sloppy 15 trillion. Quality over quantity, basically, which sounds obvious now but wasn't the default approach even three years ago.
And a third thing, which gets less attention: data mixture. It's not just how much data, but what kind, in what ratio. More code in the mix tends to help reasoning, weirdly enough, even on non-coding tasks. DeepSeek-V3 deliberately increased the share of math and code data compared to its earlier version for exactly this reason.
Architecture Tricks, Not Just "More Layers"
This is the part I find interesting, and it's where a lot of the real engineering happens. Not every improvement comes from scale. Some comes from just building the model differently.
Take Mixture-of-Experts, or MoE. Instead of one giant dense network where every parameter fires for every single token, an MoE model has a bunch of "expert" sub-networks, and a router decides which few experts actually handle a given token. DeepSeek-V3 is a good example — it has 671 billion total parameters, but only about 37 billion of them are active for any single token. That's the trick. You get the knowledge capacity of a huge model but the inference cost closer to a much smaller one.
The efficiency gains here are honestly wild. DeepSeek trained V3 for around $5.6 million, using roughly 2.79 million GPU hours reportedly less than a tenth of what a comparably-sized dense model would take. GPT-4-class training runs are estimated at $50–100 million. I'm not 100% sure those cost figures are exact (nobody outside these labs really knows the true numbers), but the gap is clearly enormous, and it's mostly architecture, not just "we found more GPUs."
DeepSeek also uses something called Multi-head Latent Attention, which compresses the memory needed to store context during inference. And in their newer V3.2 model they added sparse attention that skips computation on parts of long context that don't matter much for the current token. Every one of these is a small architectural decision, but stacked together they add up to a model that's cheaper to run and, in some benchmarks, competitive with much more expensive closed models.
I'll admit when I first read the DeepSeek papers, half the terminology went over my head. Auxiliary-loss-free load balancing, multi-token prediction, latent attention compression it's a lot. I had to read the same section three times before the MoE routing part actually clicked. But the general idea isn't that complicated once you strip the jargon: don't use the whole model for every token, and don't waste memory storing things you don't need.
Post-Training Is Where the "Personality" Comes From
Here's people miss a lot. The pretrained base model the one that just predicts the next word from raw internet text is honestly not that pleasant to talk to. It's not designed to follow instructions, refuse harmful requests, or format a nice bulleted list for you. That all comes from a separate stage called post-training.
Post-training usually has a few pieces. First there's supervised fine-tuning, where the model is shown examples of good question-answer pairs, written or reviewed by humans. Then there's reinforcement learning from human feedback, or RLHF people rank different model responses, and the model gets nudged toward the ones humans preferred. More recently, a lot of labs have shifted toward RL from AI feedback too, where another model does some of the ranking, since human ranking doesn't scale infinitely and gets expensive fast.
This is probably the biggest reason two models can have a similar number of parameters and yet feel completely different to use. Claude's tone versus GPT's tone versus Gemini's tone that's mostly a post-training decision, not a pretraining one. The base knowledge might be broadly similar; the behavior is shaped afterward.
And this is also where most of the safety work happens teaching the model when to refuse, how to handle ambiguous requests, when to ask for clarification instead of guessing. It's less flashy than "bigger model," but it's arguably where more actual research hours go these days.
The Newest Lever: Letting the Model Think Longer
Okay, this next part is probably the single biggest shift in how models improve since 2024, and it doesn't involve making the model bigger at all.
For years, an LLM answered you in roughly the same amount of compute no matter how hard the question was.Ask it "what's 2+2" or ask it to prove a tricky theorem, and it spits out tokens at the same speed either way, with the same amount of "thinking" behind each token. That's obviously wasteful for hard problems.
Reasoning models changed this. OpenAI's o1, and later o3, along with DeepSeek's R1, are trained to generate a long internal chain of reasoning before giving a final answer sometimes thousands of tokens of "thinking" the user doesn't even see directly. This is called test-time compute or inference-time compute, and it's a genuinely different lever from pretraining scale. You're not making the model's weights bigger. You're letting it use more compute at the moment you ask it a question.
The results are pretty striking on math and coding benchmarks specifically. A smaller model that's allowed to "think" for longer can beat a much bigger model that answers instantly. This is part of why DeepSeek-R1 got so much attention it showed you could get near frontier-level reasoning performance using RL-trained reasoning traces on top of a base model, without needing GPT-4-scale pretraining compute for the reasoning gains specifically.
I think this is honestly still not fully understood, even by the people building these systems. Nobody's entirely sure yet where the ceiling is on test-time compute, or how to make the "thinking" more efficient instead of just longer. There's ongoing debate about whether making a model think for 10x longer actually gives you 10x better answers, or if it plateaus fast on certain problem types. From what I've read, it plateaus a lot faster on open-ended writing tasks than on math and code, where you can automatically verify if the reasoning led somewhere correct.
So What Does an "Opus 4.5 to 4.6" Style Jump Actually Involve?
Putting it together, a version bump between models from the same family usually isn't one single change. It's a bundle:
Slightly more or better-curated pretraining data, often with a different mix maybe more code, maybe more multilingual content, maybe synthetic reasoning traces added in. Architecture tweaks that improve efficiency without necessarily growing the total parameter count that much better attention mechanisms, better routing if it's MoE, longer context windows. A refreshed post-training pass, sometimes with better human feedback data, sometimes with more RL from verifiable rewards on code and math specifically. And increasingly, better use of inference-time compute smarter"when to think longer versus when to answer fast" behavior baked in through training.
Rarely is it just "we made it 3x bigger." Pure parameter scaling has genuinely diminishing returns at this point going from 50 billion to 100 billion parameters gives you a real but shrinking improvement in loss, not a dramatic leap. The scaling curve is a power law, and power laws are stingy past a certain point. That's part of why labs have pivoted so hard toward post-training and inference-time tricks they're currently giving better returns per dollar of compute than raw pretraining scale alone.
What I'm Still Not Sure About
I don't have a clean answer for where this goes next, and I don't think anyone fully does either. The synthetic data approach worries me a little if models increasingly train on data generated by other models, does quality slowly degrade over generations, like a copy of a copy? Some researchers think this is a real risk (they call it model collapse), others think careful filtering avoids it. I've seen convincing arguments on both sides and I'm genuinely undecided.
What does seem clear is that the "just add more parameters" era is mostly behind us as the main lever, even if it's still one ingredient in the mix. The next round of improvements is probably going to come more from smarter training methods, better data curation, and figuring out how to make that inference-time thinking more efficient rather than just longer. Whether that's enough to keep the pace of improvement we've seen since 2020 that part, honestly, I don't know. Ask me again in a year.
If you're trying to actually pick a model for a project instead of just reading about how they're built, the practical advice from people tracking this closely is to stop looking at parameter count as a proxy for quality. Look at benchmark performance per dollar, and check whether the task you care about writing, coding, math, whatever actually benefits from longer reasoning time or not. Some tasks barely move with more thinking time. Others improve a lot. It depends more than most marketing pages let on.