Quantization Explained: Run 70B AI Models on Consumer Hardware

Updated 4 October 2026. This post is from April 2026 and names Llama 3.1, Qwen 2.5, Qwen 2 as current. The method and the advice still hold; today you would run it with:

• Gemma 4: use the same quantization logic for the small and medium versions to fit them into your existing RAM.
• Llama 4 Scout / Maverick: apply these same GGUF steps to the small Scout model when testing on low-memory edge hardware.
• DeepSeek V4: check the larger parameter counts against your GPU memory limits before pulling the tag.

The question we hear most from prospective clients is how a model with 70 billion parameters can run on a box that fits on a desk. The answer is quantization. This family of compression techniques cuts a model’s memory footprint by 4 to 8 times while keeping most of its quality. By the end of this post, you will know which level to pick, what fits in the machine you already have, and the exact Ollama command to pull it.

What you need

• Any computer with 8 GB of RAM or more. A laptop is enough for a 3B model; 16 GB runs 7B to 14B models; 48 GB or more runs a 70B model.
• Ollama installed (curl -fsSL https://ollama.com/install.sh | sh on Linux, brew install ollama on macOS).
• Ten minutes and about 5 GB of free disk for the first model.

What quantization does

A standard model stores each parameter as a 16-bit floating-point number (FP16). A 70B model at FP16 needs 140 GB of memory, beyond any consumer device.

Quantization lowers the precision of those numbers: 8 bits per parameter halves the memory, 4 bits quarters it, and 2-bit formats go further still. The model gets smaller, faster and cheaper to run, with a quality loss that is small at 4 bits and noticeable below that.

xychart-beta
    title "70B model: memory by quantization level"
    x-axis ["FP16 (full)", "INT8", "Q6_K", "Q5_K_M", "Q4_K_M", "Q3_K_M", "Q2_K"]
    y-axis "Memory (GB)" 0 --> 150
    bar [140, 70, 56, 48, 40, 35, 25]

At Q4_K_M (4-bit, medium quality) a 70B model drops from 140 GB to about 40 GB, which fits a Mac mini M4 Pro with 48 GB of unified memory or a Mac Studio.

The three formats that matter in 2026

GGUF (what Ollama uses)

GGUF is the format used by llama.cpp and Ollama. It is the standard for consumer hardware because it supports CPU + GPU hybrid inference: the model loads partly into GPU memory and partly into system RAM. A GPU with only 8 GB of VRAM can still hold the compute-heavy layers while the rest sits in RAM. On Apple Silicon the CPU and GPU share one memory pool, which is why Ollama on a Mac is so simple.

Start with Q4_K_M. If the output is not good enough for your task, step up to Q5_K_M. In our own use Q4_K_M has been hard to tell apart from full precision on routine business tasks such as summaries, extraction, classification and drafting; check it on yours (step 4).

AWQ (production GPU inference)

AWQ (Activation-Aware Weight Quantization) measures which weights matter most during real inference and protects those from aggressive compression. Its authors report better accuracy at 4 bits than plain round-to-nearest quantization. Major model families ship AWQ checkpoints on Hugging Face, and servers such as vLLM and TensorRT-LLM include optimised AWQ kernels. Best for dedicated GPU servers where throughput matters.

GPTQ (batch processing)

GPTQ runs a small calibration dataset through the model once to choose the quantization parameters. It suits batch jobs where latency is not critical: offline document processing, queued API requests.

How much quality you lose

The quality percentages are rough figures reported in comparison articles by Prem AI, which sells an AI platform, and VRLA Tech, which sells AI workstations. They are not a controlled benchmark and vary by model and task; treat them as an ordering, not a measurement. For summarisation, Q&A, classification and code generation the difference between Q4_K_M and full precision is hard to notice. Where it shows: long multi-step reasoning and nuanced creative writing.

Step 1: check what fits in your memory

Rule of thumb: model file size plus 1 to 2 GB for the context window, plus whatever your operating system needs. Leave a quarter of the memory free.

Step 2: pull the right tag

Ollama’s default tag for almost every model is Q4_K_M; you never touch the quantization tooling yourself. To choose another level, name it in the tag:

ollama pull llama3.3:70b            # default tag = Q4_K_M, ~43 GB
ollama pull llama3.3:70b-q5_K_M     # ~50 GB, higher quality
ollama pull llama3.3:70b-q8_0       # ~75 GB, near-original
ollama pull qwen2.5:7b              # 4.7 GB, fits a 16 GB laptop

Step 3: verify what you got

ollama show qwen2.5:7b     # prints parameters, context length and "quantization Q4_K_M"
ollama run qwen2.5:7b "Summarise this in one line: quantization trades precision for memory."
ollama ps                  # shows the real memory in use while the model is loaded

If ollama ps shows the model split between CPU and GPU (for example 60%/40% CPU/GPU), the model is too large for your GPU memory and generation will be slow; drop one level or pick a smaller model.

Step 4: decide with a measurement, not a table

Run your own task through Q4_K_M and Q5_K_M and compare the answers side by side. Ask your AI assistant to grade the two outputs against a rubric you write. If you cannot tell them apart, keep Q4_K_M and spend the memory on a longer context.

What we measured

On our NVIDIA GB10 workstation (128 GB unified memory), ollama list and ollama ps on 2026-09-09:

The last row is the point of the whole post: a 73-billion-parameter model that would need about 147 GB at FP16 sits in 48 GB on disk at Q4_K_M. Memory in use grows with the context window, and by more than most people expect: the same 4.9 GB Llama 3.1 8B file occupied 22 GB when ollama run opened it with its 131K-token default context. Set num_ctx to what your task needs (4K to 8K for most business work) and the footprint drops to a little over the file size.

The 2026 production stack

1. Discovery: LM Studio, a GUI for browsing and testing models.
2. Development and SME deployment: Ollama with GGUF Q4_K_M, the simplest path, works everywhere.
3. High-throughput production: vLLM with AWQ for API servers that need maximum requests per second.

For most SMEs, step 2 is where a deployment can stay: Ollama with a Q4_K_M model covers a small office as well as a mid-sized team.

Where this fits

Quantization is what makes the local economics work: a Mac mini or a used mini PC runs models that handle most day-to-day business tasks, electricity is a few euros a month, and your data never leaves the building (GDPR by design). The honest limit: a quantized 7B model is not a frontier model. For difficult reasoning keep a cloud key with a spending cap and route the routine volume locally; the numbers are in Cloud vs local: calculate your break-even.

Next steps

• Choose the machine: Edge AI hardware guide 2026.
• Choose the model: Best local LLM models Q2 2026.
• Go further: Fine-tune a model on your own hardware with LoRA.

Work with us

We size the model and the machine by measuring, not by guessing. If you want to see your own task running on a quantized model on real hardware, book a 15-minute call or see how we work in consulting.