Introduction: Why Self-Hosting AI is the Future
The days of depending entirely on cloud APIs for AI are ending. What once required a $100+ monthly subscription to ChatGPT or Claude now runs on your own hardware with better privacy, zero recurring costs, and complete control over which models you deploy.
Today’s open-source large language models — particularly Llama 3, DeepSeek R1, and Mistral — are sophisticated enough to handle real work. The catch? You need to know exactly what hardware keeps them running efficiently. This guide reveals the precise specifications required for optimal cost-to-performance ratios, whether you’re building a budget home lab or a high-performance Proxmox server.
The shift to self-hosting isn’t just about money. It’s about independence. Your data never leaves your network. Your model responds instantly without cloud latency. You choose which versions of which models to run. For developers, DevOps engineers, and AI enthusiasts, this freedom justifies the hardware investment.
Understanding the Core Requirements: The AI Hardware Stack
The LLM Hardware Stack: What Component Matters Most?
Running LLMs locally depends on a specific chain of components, but one dominates every decision you’ll make: the GPU’s VRAM (video memory). Think of it as the model’s working memory.
Every LLM exists as a massive parameter file — a Llama 3 7B model weighs roughly 3.5 GB in full precision. The model must load entirely into VRAM for reasonably fast inference. If it doesn’t fit, your system falls back to CPU processing and system RAM, which is hundreds of times slower. That’s the fundamental constraint shaping every build recommendation in this guide.
GPU (Graphics Card): The Crucial Factor
Your GPU is the brain of LLM inference. Everything depends on VRAM capacity. Here’s the harsh reality: VRAM is approximately 90% of the performance battle for LLM inference. This isn’t hyperbole.
Why VRAM dominates: When you load an LLM, the entire parameter file must fit in GPU memory. A 7 billion-parameter model in full 16-bit precision requires 14 GB. The same model at 8-bit quantisation needs 7 GB. At 4-bit, roughly 3.5 GB. If your VRAM capacity is insufficient, the model gets split between GPU memory and system RAM.
Every parameter lookup then requires a round-trip to slower system memory, tanking your tokens-per-second rate from 10–15 down to 1–3. You’re not just slower — you’re effectively unusable.
Quantisation changes everything: Modern LLMs are almost always deployed in quantised form (reduced precision representation). A 4-bit model maintains surprising quality while cutting the memory footprint to one-quarter. An 8-bit model trades even less quality for half the memory. For consumer hardware, 4-bit quantisation is the standard for anything larger than 7 billion parameters.
VRAM Requirements by Model Size and Quantisation:
A 7 billion-parameter model in 4-bit quantisation requires roughly 3.5 to 4 GB of VRAM. At 8-bit quantisation, the same model needs about 7 GB. In full 16-bit precision, you’re looking at 14 GB — impractical for most consumer GPUs. For practical conversation, reasoning, and coding tasks, 7B models are the sweet spot for budget builds.
Moving up to 13 billion-parameter models, 4-bit quantisation brings you to 7 GB of VRAM. Eight-bit jumps to 13 GB, and full precision hits 26 GB. This is where the RTX 4060 Ti 16GB becomes relevant — it comfortably handles 13B models with room for context and prompt caching.
Real-world usage shows 13B models excel at mid-range tasks: document summarisation, more nuanced reasoning, and code generation with context awareness.
Thirty billion-parameter models hit harder: 16 GB at 4-bit quantisation, 30 GB at 8-bit, and 60 GB in full precision. Only high-end consumer GPUs like the RTX 4090 can handle this tier, and even then, you’re constrained by VRAM capacity. The 70 billion-parameter model is where consumer hardware truly struggles. At 4-bit quantisation, a 70B model consumes 35 to 40 GB of VRAM. The RTX 4090 maxes out at 24 GB, meaning you cannot fit a 70B model entirely in GPU memory without extreme measures — CPU offloading, model sharding across multiple GPUs, or quantisation below 4-bit (which degrades quality significantly).
The DeepSeek R1 671 billion-parameter model exists in a completely different universe. Full precision requires 1.3 TB of VRAM. Even at 4-bit quantisation, you’re looking at 335 GB. Extremely aggressive quantisation down to 1.58-bit can compress it to roughly 131 GB, but at the cost of noticeable quality degradation (15–25% accuracy loss).
This is data-centre scale only — not viable for consumer hardware without extreme compromises.
Practical GPU choices for 2025:
The RTX 3060 12GB remains the most affordable entry point at $200–250 used. It handles full 7B models and heavily quantised 13B variants. The ceiling is hard: 12GB limits you to either small models or extreme quantisation. If you plan to explore beyond 7B, this card will frustrate you quickly.
The RTX 3060 is technically a 2020 release, ancient by GPU standards, but for small model inference, it’s still competent. You’ll see 7 to 10 tokens per second on a Llama 3 7B model, which is perfectly usable for hobby work and learning.
The RTX 4060 Ti 16GB (starting at $499 new, widely available July 2025+) is the sweet spot for enthusiasts and serious hobbyists. Sixteen GB comfortably runs 13B models at 4-bit, with room for context and prompt caching. It’s 1.7 times faster per-GPU-core than the 3060 and consumes only 165W, making it power-efficient for sustained 24/7 deployment. Grab a used one for $250–300 if available. Performance lands around 12 to 15 tokens per second on Llama 3 13B — noticeably snappier than the budget tier, but still interactive enough for real work.
The RTX 4090 24GB is the enthusiast standard at $1,200–1,500 new. It runs Llama 3 70B at 4-bit quantisation with acceptable speed (approximately 7 to 9 tokens per second, constrained by the 24GB VRAM ceiling requiring model management). If you want to run multiple models simultaneously or handle larger context windows, this is the minimum. For serious work, the 24GB VRAM is non-negotiable. Real-world testing shows 20 to 30 tokens per second on 13B models, leaving comfortable headroom for concurrent tasks or larger batch sizes.
AMD alternatives like the RX 7800 XT 16GB and RX 6800 XT 16GB offer comparable VRAM at lower cost, but ROCm (AMD’s CUDA equivalent) support remains less mature than NVIDIA’s ecosystem.
The driver ecosystem is improving — tools like KoboldCpp and MLC-LLM now support AMD — but CUDA dominance means NVIDIA cards enjoy broader software support and faster driver updates. For a first build, NVIDIA remains the safer choice, though AMD excels for those willing to troubleshoot ROCm quirks and potentially sacrifice some bleeding-edge optimisation.
RAM (System Memory): The Safety Net
System RAM acts as an emergency overflow when GPU VRAM fills up. Sixteen GB is genuinely the minimum; it handles the OS, system overhead, and basic context. However, the moment you fill your GPU VRAM and RAM becomes the spillover buffer, and inference speed collapses. A model offloading to system RAM runs at roughly one-tenth the speed. The performance hit is not marginal — it’s catastrophic. You’re looking at 1 to 3 tokens per second instead of 10–15. For any serious LLM work, you cannot rely on RAM as your model buffer.
The practical recommendation: Thirty-two GB minimum for any serious setup. In a Proxmox environment, where you’re running VMs alongside the LLM inference engine, 32 GB becomes even more critical. Multiple VMs demand memory. Your LLM inference service demands memory. The OS demands memory. You’ll feel every GB missing if you try to skimp here.
Sixty-four GB transforms your setup into a credible multi-model platform. Two VMs, each with 32 GB, plus headroom for the host system, plus RAM-based caching for your LLM — suddenly, you can run multiple services without painful resource contention.
For 70B model inference with CPU offloading, 64–128 GB is standard. The investment in extra RAM pays dividends quickly once you start running production workloads or testing multiple models in parallel.
CPU (Processor) and Storage (SSD)
Your CPU is surprisingly unimportant for LLM inference. The GPU does nearly all the heavy lifting. An Intel Core i7 and a low-power processor like an N100 deliver essentially identical inference speeds once VRAM capacity is matched. Where CPU matters: tokenisation speed (converting text into model-readable tokens) and context length. Faster cores help marginally, but it’s not a bottleneck worth overspending on.
Reality check: Even an older Ryzen 5 5600X or a budget i5–12400 suffices. The GPU acceleration completely dwarfs CPU performance. Skip the temptation to pair your LLM setup with a high-end CPU unless you’re also using that server for other workloads. A quad-core processor at 3.5 GHz or better handles tokenisation perfectly well. Your money goes into GPU and RAM, not CPU.
Storage, however, matters more than CPU. LLM model files are large. A 70B model in 4-bit form is 35–40 GB. Multiple models easily exceed 200 GB. NVMe SSDs load these files 5–6 times faster than SATA SSDs. The difference is stark: NVMe achieves 10–20 microsecond latency and roughly 3,500 MB/s read speed.
SATA tops out at around 100 microseconds of latency and approximately 600 MB/s. For model loading, NVMe shaves 30 to 50% off initialisation time. If you’re swapping between multiple models or frequently restarting services, an NVMe drive transforms the experience from annoying to seamless.
Storage recommendation: At minimum, a 512 GB NVMe SSD for a budget build. For realistic multi-model scenarios with Proxmox, 1 TB NVMe is the practical target. The specific model matters less than capacity — WD Black SN850X and Samsung 990 Pro are reliable choices. Avoid QLC NAND (four bits per cell) if possible; TLC (three bits) offers better sustained performance for LLM workloads where you’re repeatedly reading large model files.
Recommended Builds: From Budget to Beast
Build 1: The “Ollama Starter Kit” (Budget: $300–400)
This build proves you can experiment with LLMs on a shoestring. Target VRAM: 12GB.
Components: NVIDIA RTX 3060 12GB GPU used at $200–250. Budget B450/B550 AM4 chipset motherboard. AMD Ryzen 5 5600X CPU used at $100–120, or equivalent. Thirty-two GB DDR4 RAM at $80–100. 512 GB NVMe SSD at $50–70. Six-hundred-fifty-watt 80+ Bronze PSU at $60–80.
Total cost: Approximately $550–700 (new components; cheaper if buying used ecosystem parts).
Best for: Hobby use, learning Ollama, testing 7B models, tinkering with local AI without massive investment.
Performance expectations: Seven to ten tokens per second on Llama 3 7B. Quantised 13B models are possible but sluggish. Single model loading and inference, not multi-model concurrency. This is a learning rig, not a production system.
Why this works: The RTX 3060 12GB is ancient by GPU standards (2020 release) but remains excellent for small models. System RAM is adequate for a single-model workload. No frills, no Proxmox, no VMs — just a direct Ubuntu Server installation with Ollama running the show. You get your feet wet, understand quantisation, and experience how models behave at different sizes. After a month or two here, you’ll understand whether you want to invest in a larger build.
Build 2: The “DevOps Home Lab” (Mid-Range: $800–1,200)
This is the Proxmox sweet spot. Target VRAM: 16GB GPU plus 32GB system RAM minimum.
Components: NVIDIA RTX 4060 Ti 16GB new at $499, or used RTX 3090 at $700–800 (second-hand market). Supermicro X12 series motherboard or ASUS ProArt B550 with VT-d/IOMMU support for GPU passthrough. AMD Ryzen 7 5800X3D or Intel i7–12700K CPU used at $200–250. Sixty-four GB DDR5 RAM at $200–250. One TB NVMe SSD at $80–120. Eight-hundred-fifty-watt 80+ Gold PSU at $100–150.
Total cost: Approximately $1,300–1,800 new; $900–1,200 if buying mid-generation used GPUs.
Best for: Running Proxmox with multiple VMs, hosting LLM services alongside other workloads, testing infrastructure, integration with CI/CD pipelines.
Performance expectations: Twelve to fifteen tokens per second on Llama 3 13B at 4-bit. Multiple 7B models can run concurrently with resource isolation through VMs. Smooth handling of 30B models with quantisation. Real-world testing shows this tier can host development environments, monitoring stacks, and inference services simultaneously without painful contention.
Why this build wins: Sixty-four GB RAM transforms this into a multi-workload platform. Proxmox GPU passthrough (covered in Day 2 tutorial) becomes genuinely practical. You can run a 13B model in one VM, a dev environment in another, and still maintain system stability. The RTX 4060 Ti’s power efficiency (165W) means reasonable electricity costs even in 24/7 deployment. This is where you stop tinkering and start building actual infrastructure. You begin learning Proxmox seriously, understanding resource allocation, and experiencing how production-grade VM isolation works.
Build 3: The “AI Training Powerhouse” (Expert: $2,500+)
This is for serious work: fine-tuning, large models, production inference. Target VRAM: 24GB+ GPU, 128GB+ system RAM, multi-GPU consideration.
Components: NVIDIA RTX 4090 24GB at $1,200–1,500, or multi-GPU setup with dual RTX 4090 or professional cards like RTX 6000. ASUS Pro WS Z790-SAGE UEFI motherboard with multi-PCIe slot support for multi-GPU scenarios. AMD Threadripper 5990WX or Intel Xeon W5–3435X at $500–1,000. One-hundred-twenty-eight to two-hundred-fifty-six GB DDR5 RAM at $600–1,200. Four TB NVMe SSD (multiple drives, RAID 0 for throughput) at $300–500. Sixteen-hundred-watt 80+ Platinum PSU at $300–500.
Total cost: $4,000–7,000 for high-end single-GPU; $8,000–15,000+ for multi-GPU setup.
Best for: Production LLM services, fine-tuning proprietary models, research, high-throughput inference, and multi-model concurrent serving.
Performance expectations: Twenty to thirty tokens per second on Llama 3 70B at 4-bit. DeepSeek R1 671B inference is possible with extreme quantization (1.58–2 bit), though quality degradation becomes noticeable. Fine-tuning of 13B-70B models on private datasets. Simultaneous inference of multiple 70B-equivalent models through multi-GPU parallelism and model sharding strategies.
Why this scales: The RTX 4090’s 24GB VRAM comfortably fits 70B models at 4-bit quantisation. One-hundred-twenty-eight GB system RAM enables model parallelisation strategies — sharding across multiple GPUs, redundant loading for fast model switching. High core-count CPUs (32+ cores in Threadripper) accelerate preprocessing and tokenization for batched inference. This setup approaches data-centre-grade performance on consumer hardware.
You’re no longer experimenting; you’re running a legitimate inference service that could serve a small team or power API endpoints.
Software Pre-Requisites: Bridging to Production
To operationalise any of these builds, you need specific software foundations.
Proxmox VE becomes your virtualisation hypervisor, especially in the mid-range and expert builds. It manages VM resource allocation, enables GPU passthrough (native NVIDIA driver support in guest VMs), and isolates workloads. Bare metal hypervisors require significant troubleshooting for GPU passthrough; Proxmox handles it elegantly. It’s free, open-source, and widely adopted in production environments.
Ubuntu Server is the guest OS inside your Proxmox VMs. It offers the best NVIDIA driver compatibility, stable package repositories, and wide Ollama support. CentOS/RHEL works but introduces unnecessary friction for hobbyists. Windows VMs work too, but add overhead and licensing costs — skip it unless you have a specific reason. Ubuntu Server 22.04 LTS is the standard as of 2025, offering five years of security updates.
Ollama simplifies everything. It’s a single-command tool that downloads, quantises, and serves LLMs via a REST API. Instead of wrestling with a llama.cpp, GPTQ tooling, or quantisation frameworks, you run, ollama pull llama3:13b-q4_K_M and within seconds, the model is available at localhost:11434. Ollama abstracts away the infrastructure complexity. Behind the scenes, it's optimising GPU offloading, managing context windows, and handling model loading—but you never see that complexity. It's the bridge between raw hardware acceleration and user-friendly inference.
Open WebUI wraps Ollama with a ChatGPT-like interface. If you’re not using the REST API programmatically, Open WebUI gives you a web-based front-end for chat, image understanding, and model switching. It’s optional, but it dramatically improves usability for non-technical users on your network orwhen testing different models without API calls.
CUDA Toolkit and cuDNN must be installed on the host Proxmox system (version coordination is critical). Proxmox handles GPU driver installation, but CUDA libraries enable the actual GPU compute. For NVIDIA RTX cards, CUDA 12.1+ is standard as of 2025. Older GPUs (RTX 2000/3000 series) may require CUDA 11.8. Version mismatch is a common source of “GPU not detected” errors — verify compatibility before installation. The specific version matters; wrong combinations waste hours troubleshooting.
Conclusion: The VRAM Decision is Everything
VRAM is the defining constraint of local LLM deployment. Everything else — CPU, RAM, storage, software — supports VRAM optimisation. A 12 GB GPU limits you to 7B models and heavily quantised 13B variants. A 16 GB GPU opens the 13–30B model range comfortably. A 24 GB GPU is the entry point for 70B models. This isn’t arbitrary — it’s the physics of how neural networks work. Memory access patterns dominate inference latency far more than raw compute power.
Your next step is immediate: choose your VRAM target based on which models matter to you. If you want to run Llama 3 70B locally, 24 GB is non-negotiable. If 7B reasoning models suffice, a budget RTX 3060 gets you there for under $700 total. If you want flexibility and future-proofing without breaking the bank, the RTX 4060 Ti 16GB is the best price-to-VRAM ratio in 2025.
Once hardware is locked in, the Proxmox setup begins. GPU passthrough is the bridge between raw hardware and productionized infrastructure. That’s where Day 2 of this series takes over — the tutorial on GPU passthrough configuration is the final piece enabling true local AI independence.
You’ll move from “I have hardware with a GPU” to “I have a production-grade LLM server running multiple models across isolated VMs.”
Stay tuned for the next article, don't forget to subscribe for notifications!!