I spent six months running LLMs on gaming hardware before I learned the hard truth. That shiny RTX 4090 sitting in my desktop was burning electricity, throttling under sustained load, and still could not handle the models I actually needed for production work. The specifications looked impressive on paper. Twenty-four gigabytes of VRAM is plenty. But specifications tell you nothing about what happens when you run inference 24/7 for weeks.
Professional AI hardware exists for reasons that only become clear when you try to deploy real workloads. This is not about benchmarks or theoretical performance. This is about what actually works when you need to serve hundreds of users, maintain uptime guarantees, and sleep at night knowing your infrastructure will not corrupt model weights halfway through a training run.
When Gaming Cards Fall Short
Memory bandwidth is the specification nobody talks about until it destroys your project timeline. The RTX 4090 delivers one terabyte per second of memory bandwidth. That sounds fast until you compare it to an NVIDIA H100 pushing 3.35 terabytes per second. LLM inference is not compute-bound. Your GPU spends most of its time waiting for memory to deliver the next parameter. Faster computation means nothing when the bottleneck is fetching weights from memory.
The performance difference shows up immediately in tokens per second. A 4090 running Llama 3 70B at 4-bit quantisation generates maybe seven to nine tokens per second. An H100 running the identical model configuration produces forty to fifty tokens per second. That is not a 10 percent improvement. That is five times faster. When you are serving real users who expect sub-second response times, this difference determines whether your service is viable.
Gaming cards throttle under sustained workloads because manufacturers design them for burst performance. You game for a few hours, the card heats up, fans spin louder, performance drops slightly, then you stop and everything cools down. AI training runs for days. Inference services run continuously for months. Gaming cards were never engineered for this usage pattern. Thermal throttling becomes a constant battle rather than an occasional inconvenience.
Error correction matters more than anyone expects. Gaming GPUs lack ECC memory entirely. Every bit flip in memory goes undetected and potentially corrupts your model weights. When you have seventy billion parameters, even rare errors accumulate. Your model starts producing nonsensical outputs and you have no idea whether the problem is your training data, your prompting strategy, or silent memory corruption. Data center GPUs implement ECC as standard. The hardware detects errors, corrects them automatically, and logs the incident. This is not optional for production deployments.
NVIDIA A100: The Workhorse That Refuses to Die
The A100 launched in 2020, and everyone assumed newer hardware would replace it quickly. Four years later, A100S still dominate production AI infrastructure. There are good reasons for this persistence. The eighty-gigabyte variant hits the sweet spot for most real-world LLM deployments. You can run 70B models comfortably at 4-bit quantisation with substantial context window capacity. That covers probably 80 per cent of actual production use cases.
Multi-Instance GPU capability is the feature that keeps A100S relevant despite newer alternatives. You can partition a single A100 into up to seven independent GPU instances, each with isolated memory and compute resources. One instance serves your customer-facing chatbot running a 13B model. Another instance handles batch document processing. A third instance runs experimental fine-tuning jobs. All three workloads run simultaneously on the same physical GPU with zero interference.
The economics make sense in ways that newer hardware does not. Used A100 80GB cards trade around $12,000 as organisations upgrade to H100S. That price point delivers credible production capability without the sticker shock of current-generation hardware. You build a dual-A100 server for roughly $30,000 total system cost and suddenly, you have enough capacity to serve thousands of users daily. The math works.
Performance is adequate rather than exciting. Fifteen to twenty tokens per second on 70B models feels slow compared to H100 benchmarks. But adequate performance at one-third the cost beats exceptional performance you cannot afford. Most production deployments care more about cost per token than absolute speed. A100 delivers the lowest cost per token for 70B-class models among hardware you can actually purchase today.
The used market availability is the hidden advantage. H100S have months-long waitlists even at inflated prices. A100S are available now from organisations that already upgraded. You can build infrastructure this quarter rather than waiting until next year. For startups and mid-size companies, time-to-deployment often matters more than having the absolute fastest hardware.
NVIDIA H100: When You Need Maximum Performance
H100 is what you buy when performance directly impacts revenue. The Transformer Engine optimisation specifically targets attention mechanisms in modern LLMs. Attention operations typically consume 60 to 80 per cent of computation time in transformer models. H100 processes attention roughly 2.5 times faster than A100 through architectural improvements that understand transformer workloads at the hardware level.
Memory bandwidth is where H100 pulls away from everything else. That 3.35 terabytes per second of bandwidth translates directly to inference speed. You are not waiting for parameters to load. The model weights flow continuously from HBM memory into tensor cores with minimal latency. This matters enormously for latency-sensitive applications where every millisecond counts.
NVLink connectivity becomes critical when you need to run models larger than eighty gigabytes. A 405 billion-parameter model at 4-bit quantisation requires roughly two hundred gigabytes of VRAM. No single GPU provides that capacity. Three H100S connected via NVLink can distribute the model across their combined memory with minimal communication overhead. NVLink 4.0 provides nine hundred gigabytes per second of bidirectional bandwidth between GPUs. The model shards across multiple GPUs but still delivers usable inference speeds around eight to twelve tokens per second.
The pricing reflects enterprise positioning rather than hobbyist accessibility. H100 80GB cards sell for $35,000 when you can find them. New purchases through NVIDIA partners exceed $40,000 per card with substantial lead times. Cloud providers charge four to six dollars per hour for H100 instances. This pricing structure targets organisations where faster inference directly generates revenue or competitive advantage.
The calculation only makes sense for specific scenarios. You are running extremely large models that require multi-GPU deployments. You are serving high-throughput production inference where response time directly impacts user experience and, therefore, revenue. You are conducting research requiring the fastest possible training cycles. For most organisations running 70B or smaller models, the H100 cost cannot be justified against A100 or AMD alternatives.
AMD MI300X: The Memory Capacity Champion
AMD finally built something that competes directly with NVIDIA at the high end. The MI300X ships with 192 gigabytes of HBM3 memory per accelerator. That capacity enables deployment scenarios that NVIDIA hardware cannot match at any price. A single MI300X runs 405 billion-parameter models at 4-bit quantisation with comfortable headroom for extended context windows. Models approaching a trillion parameters become feasible with multi-GPU setups, where each GPU contributes to that massive memory capacity.
Memory bandwidth exceeds H100 specifications. The MI300X delivers 5.3 terabytes per second compared to H100’s 3.35 terabytes per second. Since LLM inference is overwhelmingly memory-bound, this bandwidth advantage translates directly to faster token generation. Independent testing shows MI300X achieving forty-five to sixty tokens per second on Llama 3 70B at 4-bit quantisation. That matches or exceeds H100 performance depending on specific model architecture and optimisation quality.
The pricing is where AMD becomes genuinely interesting. MI300X cards sell for $18,000 to $25,000 when available. That is substantially below H100’s $35,000 to $40,000 range. Cloud instances typically price twenty to thirty per cent cheaper than equivalent H100 configurations. For organisations that prioritise raw memory capacity and can work within ROCm constraints, MI300X delivers better economics than any NVIDIA alternative.
ROCm software ecosystem is the persistent challenge that keeps many organisations on NVIDIA despite AMD’s hardware advantages. ROCm provides AMD’s equivalent to CUDA, but driver stability and optimization lag NVIDIA by roughly twelve to eighteen months. Major LLM frameworks like PyTorch and vLLM now support ROCm in recent releases. But edge cases exist. Documentation is less comprehensive. Fewer engineers have ROCm troubleshooting experience.
The practical decision comes down to team capability and risk tolerance. Organisations with strong infrastructure teams comfortable debugging driver issues, find ROCm’s improved economics justify the friction. Teams prioritising rapid deployment and minimal operational overhead stick with NVIDIA despite the premium. ROCm has reached the point where large production deployments are viable with appropriate engineering support. It is not there yet for organisations wanting plug-and-play deployment.
The Real TCO Calculation Nobody Does
Hardware acquisition cost is only the beginning of the total cost of ownership. An A100 80GB at $12,000 requires server infrastructure around it. You need a motherboard, CPU, RAM, power supply, cooling, and rack space, totalling another $4,000 to $6,000. The complete system costs $16,000 to $18,000 before you power it on.
Electricity matters more than people expect. Four hundred watts of continuous power draw costs approximately $350 per year at typical US electricity rates around $0.10 per kilowatt-hour. Over a three-year deployment, you spend roughly $1,000 just on electricity for a single GPU. That sounds small compared to acquisition cost but it compounds across multiple GPUs and multiple years.
Cooling costs are the hidden expense that catches organisations unprepared. That four hundred watts of heat needs to go somewhere. Data centre cooling typically doubles your power consumption. You draw four hundred watts for the GPU and another four hundred watts to cool it. Now your electricity cost just doubled. Organisations deploying AI infrastructure in office environments discover they need dedicated cooling solutions they never budgeted for.
Maintenance and replacement costs hit you eventually. GPUs fail. Power supplies die. SSDs wear out. Budget roughly ten to fifteen percent of hardware cost annually for maintenance and component replacement. Over three years on that $18,000 A100 system, maintenance costs add another $5,000 to $8,000.
Cloud deployment eliminates upfront costs but charges you continuously. H100 instances at five dollars per hour cost $43,800 annually at full utilisation. For intermittent workloads where you use compute a few hours daily, cloud pricing is dramatically cheaper. For sustained 24/7 deployment, hardware purchase pays for itself in four to six months. The break-even calculation is straightforward. Below three thousand hours annually, cloud wins. Above five thousand hours, on-premises hardware becomes economical.
What Actually Matters for Production Deployment
Memory capacity determines which models you can run. Everything else is optimisation around that core constraint. NVIDIA A100 80GB handles 70B models comfortably. H100 with 94GB provides headroom for extended context or multiple model loading. AMD MI300X with 192GB enables 405B models on a single accelerator.
Calculate your target model size, multiply parameters by quantisation bits per parameter, and ensure your hardware provides at least 1.5 times that capacity to account for context and operational overhead.
Software ecosystem maturity determines how much time your team spends on infrastructure versus building applications. NVIDIA’s CUDA ecosystem has a decade of refinement. Every framework prioritises NVIDIA support. When issues arise, solutions exist on Stack Overflow and GitHub. AMD’s ROCm improves with every release but trails CUDA in maturity. You spend more engineering time troubleshooting and less time shipping features.
Batch inference capability matters for production services serving many concurrent users. H100 excels at batch inference with proper tuning, achieving three to four times higher throughput when batching thirty-two or sixty-four requests together. This advantage matters enormously for API services handling hundreds of concurrent users. For single-user scenarios like coding assistants, batch advantages disappear, and raw memory bandwidth dominates.
Thermal characteristics affect where you can actually deploy hardware. Gaming cards sound like jet engines under sustained load. Data centre GPUs with proper cooling run quietly and maintain consistent performance indefinitely. This difference determines whether you can deploy AI infrastructure in office environments or need dedicated server rooms with industrial cooling.
The decision framework is straightforward. Determine which models your applications require. Calculate memory footprint at appropriate quantisation levels. Select hardware providing adequate capacity with acceptable bandwidth and software support.
Everything else is details. The professional AI hardware market now provides options across the entire spectrum from modest deployments to data centre scale.
Understanding these trade-offs lets you choose the right tool for your specific requirements rather than defaulting to the most expensive option.