I wanted to know, concretely, what you lose when you take a 70B-parameter model down to 4-bit and squeeze it onto a single 3090. Not benchmark-suite numbers, which tend to wash out the differences that matter in practice, but whether the model's answers on my own tasks got worse in a way I'd notice day to day.

Short version: for most conversational and coding tasks, 4-bit AWQ was close enough to full precision that I stopped noticing a difference within a day of use. The gap shows up in specific places — multi-step arithmetic, precise instruction-following with many constraints stacked in one prompt, and anything requiring the model to hold a long chain of exact facts without drifting. Those are exactly the tasks where quantization noise compounds across a long generation, and it shows.

GGUF vs AWQ, in practice

GGUF with llama.cpp is what I'd point a hobbyist toward — the tooling is mature, CPU offload for whatever doesn't fit in VRAM actually works instead of just existing on paper, and the quantization schemes below 4-bit (like the K-quants) hold up better than I expected for casual use. AWQ is the one I reach for when I actually need throughput, since it plays nicer with batched serving through vLLM. If you're serving more than one request at a time, that difference matters more than the marginal quality gap between the two formats.

quantization  | vram (70B) | tok/s (single stream, 3090) | notable degradation
fp16          | ~140GB     | n/a (doesn't fit)           | baseline
8-bit         | ~70GB      | n/a (still doesn't fit)     | negligible
4-bit AWQ     | ~38GB      | ~18                          | multi-step math, long exact recall
4-bit GGUF Q4_K_M | ~40GB  | ~14 (partial CPU offload)    | similar, slightly more variance

Those numbers are from my own setup, not a rigorous benchmark — take them as "roughly what to expect," not a citation. The point isn't the exact tokens-per-second, it's that even a single consumer card gets you a genuinely capable local model now, with degradation that's specific and predictable rather than a uniform quality hit across the board.

Where I'd actually use this

Anything where data can't leave the machine, or where you're iterating fast enough that API costs and rate limits start to matter. I wouldn't reach for a locally quantized 70B for a task that leans hard on precise multi-step reasoning — for that I'd rather pay for a hosted frontier model and eat the latency and cost. Know which bucket your task falls into before you commit to a local setup; guessing wrong after you've already built the pipeline is a more expensive mistake than it needs to be.