GPU compatibility

What LLMs can run on 24 GB VRAM homelab workstation?

Heavy local models and homelab inference. Examples: RTX 3090, RTX 4090.

Open calculator with this GPU preset

Clean local fits26

Offload / slower7

Too large1

Planning capacity22.5 GB clean VRAM

Recommended route: local-first

This preset has enough clean Q4 headroom for most curated local models in this dataset.

Use locally forcoding assistants, chat, routing, agents, and larger 14B-32B class models depending on context length.

Watch out forvery large models, long context windows, and parallel workloads can still exceed practical memory.

Fallback triggerUse cloud/API when you need 70B+ models, long context, or reliable multi-user throughput.

Green

26 models fit inside the clean planning capacity.

Yellow

7 models can run with RAM/offload tradeoffs.

Red

Examples to avoid locally: GLM-5.2.

Best clean fits

Start here for responsive local inference.

26 models

Qwen3 4B Thinking 2507

4.02B Runs locally Q4 about 3.2 GB

Small local reasoning, routing, tool decisions, and lightweight coding on 6GB-8GB GPUs

Best local fit Open model page ->

Gemma 3 4B

4B Runs locally Q4 about 3.5 GB

Small multimodal local assistant and low-resource setups

Best local fit Open model page ->

Qwen2.5 Coder 7B

7B Runs locally Q4 about 5.5 GB

Small local coding assistant and agent tool generation

Best local fit Open model page ->

Mistral 7B

7B Runs locally Q4 about 5.5 GB

Fast local chat and simple agent tasks

Best local fit Open model page ->

Possible with offload

Use only when slower generation and tighter settings are acceptable.

7 models

Qwen3.6 35B-A3B

35B RAM offload Q4 about 24 GB

Agentic coding and multimodal reasoning when 27B is not enough and 32GB-class headroom is available

Possible, slower Open model page ->

Mixtral 8x7B

46.7B RAM offload Q4 about 28 GB

MoE local reasoning/chat when enough VRAM is available

Possible, slower Open model page ->

Llama 3.1 70B Instruct

70B RAM offload Q4 about 44 GB

High-quality local chat and reasoning on workstation-class hardware

Possible, slower Open model page ->

Qwen3-Next 80B-A3B Instruct

80B RAM offload Q4 about 50 GB

High-end local reasoning, long-context planning, and tool-agent workloads when 48GB+ memory is available

Possible, slower Open model page ->

Full Q4 model fit table

Model	Size	Q4 need	Status	Calculator
Qwen3 4B Thinking 2507	4.02B	3.2 GB	Runs locally	Open calculator
Gemma 3 4B	4B	3.5 GB	Runs locally	Open calculator
Qwen2.5 Coder 7B	7B	5.5 GB	Runs locally	Open calculator
Mistral 7B	7B	5.5 GB	Runs locally	Open calculator
Llama 3.1 8B Instruct	8B	6 GB	Runs locally	Open calculator
Qwen3 8B	8B	6 GB	Runs locally	Open calculator
DeepSeek-R1-0528-Qwen3-8B	8B	6 GB	Runs locally	Open calculator
Qwen3-VL 8B Instruct	8B	6.5 GB	Runs locally	Open calculator
Qwen3.5 9B	9B	6.6 GB	Runs locally	Open calculator
Gemma 3 12B	12B	9 GB	Runs locally	Open calculator
Gemma 4 E4B	4B	9.6 GB	Runs locally	Open calculator
Qwen2.5 Coder 14B	14B	10.5 GB	Runs locally	Open calculator
DeepSeek R1 Distill Qwen 14B	14B	10.5 GB	Runs locally	Open calculator
Phi-4 14B	14B	10.5 GB	Runs locally	Open calculator
gpt-oss 20B	20B	14 GB	Runs locally	Open calculator
Devstral Small 2 24B	24B	15 GB	Runs locally	Open calculator
Qwen3.5 27B	27B	17 GB	Runs locally	Open calculator
Qwen3.6 27B	27B	17 GB	Runs locally	Open calculator
Gemma 3 27B	27B	18 GB	Runs locally	Open calculator
Qwen3 30B-A3B Instruct 2507	30.5B	19 GB	Runs locally	Open calculator
Qwen3-Coder 30B-A3B	30B	19 GB	Runs locally	Open calculator
GLM-4.7-Flash	30B	19 GB	Runs locally	Open calculator
Gemma 4 31B	31B	20 GB	Runs locally	Open calculator
Qwen3-VL 30B-A3B Instruct	30B	20 GB	Runs locally	Open calculator
Qwen2.5 Coder 32B	32B	21 GB	Runs locally	Open calculator
DeepSeek R1 Distill Qwen 32B	32B	21 GB	Runs locally	Open calculator
Qwen3.6 35B-A3B	35B	24 GB	RAM offload	Open calculator
Mixtral 8x7B	46.7B	28 GB	RAM offload	Open calculator
Llama 3.1 70B Instruct	70B	44 GB	RAM offload	Open calculator
Qwen3-Next 80B-A3B Instruct	80B	50 GB	RAM offload	Open calculator
Qwen3-Coder-Next	80B	52 GB	RAM offload	Open calculator
gpt-oss 120B	120B	65 GB	RAM offload	Open calculator
Devstral 2 123B	123B	75 GB	RAM offload	Open calculator
GLM-5.2	744B	466 GB	Too large	Open calculator