GPU compatibility

What LLMs can run on 12 GB VRAM local agent GPU?

Local routing, agents, and model testing. Examples: RTX 3060 12GB, RTX 4070, RTX 5070.

Open calculator with this GPU preset

Clean local fits14
Offload / slower14
Too large6
Planning capacity10.5 GB clean VRAM

Recommended route: local with limits

This preset is useful locally, but model choice matters. Stay close to the green list for the best experience.

Use locally forsmall chat models, coding helpers, Telegram bots, routing layers, and selected 7B-14B Q4 models.
Watch out foryellow models may run slowly because they need RAM/offload or reduced context settings.
Fallback triggerUse hosted API or cloud GPU when the chosen model lands in yellow for production work.
Green

14 models fit inside the clean planning capacity.

Yellow

14 models can run with RAM/offload tradeoffs.

Red

Examples to avoid locally: Llama 3.1 70B Instruct, Qwen3-Next 80B-A3B Instruct, Qwen3-Coder-Next.

Best clean fits

Start here for responsive local inference.
14 models

Qwen3 4B Thinking 2507

4.02B Runs locally Q4 about 3.2 GB

Small local reasoning, routing, tool decisions, and lightweight coding on 6GB-8GB GPUs

Best local fit Open model page ->

Gemma 3 4B

4B Runs locally Q4 about 3.5 GB

Small multimodal local assistant and low-resource setups

Best local fit Open model page ->

Qwen2.5 Coder 7B

7B Runs locally Q4 about 5.5 GB

Small local coding assistant and agent tool generation

Best local fit Open model page ->

Mistral 7B

7B Runs locally Q4 about 5.5 GB

Fast local chat and simple agent tasks

Best local fit Open model page ->

Possible with offload

Use only when slower generation and tighter settings are acceptable.
14 models

gpt-oss 20B

20B RAM offload Q4 about 14 GB

Local reasoning, agent planning, and tool-use workflows on 16GB+ GPUs

Possible, slower Open model page ->

Devstral Small 2 24B

24B RAM offload Q4 about 15 GB

Software engineering agents, repo navigation, patch planning, and local coding workflows

Possible, slower Open model page ->

Qwen3.5 27B

27B RAM offload Q4 about 17 GB

24GB-class multimodal agent, coding assistant, and reasoning workloads

Possible, slower Open model page ->

Qwen3.6 27B

27B RAM offload Q4 about 17 GB

Newest 27B-class local multimodal coding, agent, and reasoning workloads on 24GB-class GPUs

Possible, slower Open model page ->

Full Q4 model fit table

ModelSizeQ4 needStatusCalculator
Qwen3 4B Thinking 25074.02B3.2 GBRuns locallyOpen calculator
Gemma 3 4B4B3.5 GBRuns locallyOpen calculator
Qwen2.5 Coder 7B7B5.5 GBRuns locallyOpen calculator
Mistral 7B7B5.5 GBRuns locallyOpen calculator
Llama 3.1 8B Instruct8B6 GBRuns locallyOpen calculator
Qwen3 8B8B6 GBRuns locallyOpen calculator
DeepSeek-R1-0528-Qwen3-8B8B6 GBRuns locallyOpen calculator
Qwen3-VL 8B Instruct8B6.5 GBRuns locallyOpen calculator
Qwen3.5 9B9B6.6 GBRuns locallyOpen calculator
Gemma 3 12B12B9 GBRuns locallyOpen calculator
Gemma 4 E4B4B9.6 GBRuns locallyOpen calculator
Qwen2.5 Coder 14B14B10.5 GBRuns locallyOpen calculator
DeepSeek R1 Distill Qwen 14B14B10.5 GBRuns locallyOpen calculator
Phi-4 14B14B10.5 GBRuns locallyOpen calculator
gpt-oss 20B20B14 GBRAM offloadOpen calculator
Devstral Small 2 24B24B15 GBRAM offloadOpen calculator
Qwen3.5 27B27B17 GBRAM offloadOpen calculator
Qwen3.6 27B27B17 GBRAM offloadOpen calculator
Gemma 3 27B27B18 GBRAM offloadOpen calculator
Qwen3 30B-A3B Instruct 250730.5B19 GBRAM offloadOpen calculator
Qwen3-Coder 30B-A3B30B19 GBRAM offloadOpen calculator
GLM-4.7-Flash30B19 GBRAM offloadOpen calculator
Gemma 4 31B31B20 GBRAM offloadOpen calculator
Qwen3-VL 30B-A3B Instruct30B20 GBRAM offloadOpen calculator
Qwen2.5 Coder 32B32B21 GBRAM offloadOpen calculator
DeepSeek R1 Distill Qwen 32B32B21 GBRAM offloadOpen calculator
Qwen3.6 35B-A3B35B24 GBRAM offloadOpen calculator
Mixtral 8x7B46.7B28 GBRAM offloadOpen calculator
Llama 3.1 70B Instruct70B44 GBToo largeOpen calculator
Qwen3-Next 80B-A3B Instruct80B50 GBToo largeOpen calculator
Qwen3-Coder-Next80B52 GBToo largeOpen calculator
gpt-oss 120B120B65 GBToo largeOpen calculator
Devstral 2 123B123B75 GBToo largeOpen calculator
GLM-5.2744B466 GBToo largeOpen calculator