GPU compatibility

What LLMs can run on 12 GB VRAM local agent GPU?

Local routing, agents, and model testing Examples: RTX 3060 12GB, RTX 4070.

Open calculator with this GPU preset

Preset VRAM12 GB
System memory32 GB
Use caseLocal routing, agents, and model testing

Best clean fits

Gemma 3 4B

Small multimodal local assistant and low-resource setups

Qwen2.5 Coder 7B

Small local coding assistant and agent tool generation

Mistral 7B

Fast local chat and simple agent tasks

Llama 3.1 8B Instruct

Fast local chat, lightweight agents, low-cost local testing

Q4 model fit table

ModelSizeQ4 needStatusCalculator
Gemma 3 4B4B3.50 GBRuns locallyOpen calculator
Qwen2.5 Coder 7B7B5.50 GBRuns locallyOpen calculator
Mistral 7B7B5.50 GBRuns locallyOpen calculator
Llama 3.1 8B Instruct8B6.00 GBRuns locallyOpen calculator
Qwen3 8B8B6.00 GBRuns locallyOpen calculator
DeepSeek R1 Distill Qwen 8B8B6.00 GBRuns locallyOpen calculator
Gemma 3 12B12B9.00 GBRuns locallyOpen calculator
Qwen2.5 Coder 14B14B10.5 GBRuns locallyOpen calculator
DeepSeek R1 Distill Qwen 14B14B10.5 GBRuns locallyOpen calculator
Phi-4 14B14B10.5 GBRuns locallyOpen calculator
Gemma 3 27B27B18 GBRAM offloadOpen calculator
Qwen2.5 Coder 32B32B21 GBRAM offloadOpen calculator
DeepSeek R1 Distill Qwen 32B32B21 GBRAM offloadOpen calculator
Mixtral 8x7B46.7B28 GBRAM offloadOpen calculator
Llama 3.1 70B Instruct70B44 GBToo largeOpen calculator