Running LLMs on a Single GPU — The Practical Playbook

The single most common misconception about local AI is that you need a rack of A100s. You don’t. A single consumer GPU — even an 8 GB card — can run genuinely useful models today, and a 16–24 GB card (like the RTX 4080) can run models that hold their own against hosted APIs for most everyday work.

This guide is the playbook I wish I’d had: what actually matters, what to install, and how to pick a model that fits your VRAM.

The only number that matters: VRAM

Everything else — TOPS, TFLOPS, memory bandwidth — matters, but the hard constraint is memory. A model’s size in VRAM is roughly:

VRAM needed ≈ parameter count × bytes per weight × 1.15 (overhead)

Bytes per weight depends on quantisation:

Format	Bits per weight	7B model	13B model	70B model
FP16	16	~14 GB	~26 GB	~140 GB
Q8_0	8	~7.5 GB	~14 GB	~75 GB
Q4_K_M	4.5	~4.5 GB	~8.1 GB	~40 GB
Q2_K	2.6	~3 GB	~5 GB	~25 GB

Rule of thumb: Q4_K_M is the sweet spot — roughly 90% of FP16 quality at a third of the memory. If the model barely fits at Q4_K_M, it won’t be comfortable; drop a size class instead of dropping quality further.

The stack that just works

# 1. Ollama — the easiest path, one binary
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3:14b-q4_K_M

# 2. llama.cpp — for the most control (and CPU+GPU offload)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

# 3. GPU info that matters
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

Ollama is the best default: sensible defaults, OpenAI-compatible API at http://localhost:11434, and it manages downloads. llama.cpp is for when you want exact control over layers offloaded, context length, and prompt caching.

Picking a model that fits your card

GPU	VRAM	Best models (Q4_K_M)
RTX 4060 / 3060	8 GB	3–8B: Qwen3 8B, Llama 3.2 3B
RTX 4070 Ti / 3070	12 GB	8–14B: Qwen3 14B, Llama 3.1 8B
RTX 4080 / 4090	16–24 GB	14–32B: Qwen3 32B, Mistral Small 24B
2× 3090	48 GB	70B: Llama 3.3 70B (slow but works)

For coding, smaller models with big context beat bigger models with small context. For reasoning, anything under 14B starts to struggle with multi-step problems — that’s where Qwen3’s thinking mode earns its keep.

Making it actually fast

Three settings matter more than raw tokens-per-second:

1. --ctx-size 8192 minimum. Models are useless if they forget the conversation. 32k if your VRAM allows — KV cache grows with context, and that’s why a 7B at 32k context can use more VRAM than a 13B at 4k.
2. --n-gpu-layers — offload everything. Set it to the max your VRAM fits; CPU layers become the bottleneck instantly. Ollama does this automatically.
3. Keep-alive. A warm model in VRAM responds in ~100 ms; a cold model takes 5–15 s to load. ollama serve keeps the last model resident — batch your usage.

When to use the cloud instead

Local models shine for: privacy (medical/legal/financial text), offline work, high-volume repetitive tasks, and anything where the capability ceiling of frontier models matters more than cost. They struggle with: top-tier creative writing, the hardest reasoning benchmarks, and enormous contexts.

The pragmatic stack is both: a local 14–32B for daily work, a hosted frontier model for the weekly hard problems. Set up OpenRouter with a $5 balance and you’ll never think about it again.

Your first move

ollama pull qwen3:14b-q4_K_M
ollama run qwen3:14b-q4_K_M

Ask it something genuinely hard, then check nvidia-smi while it’s thinking. You’re now part of the local-AI revolution — no data centre required.