GPU compatibility

What LLMs can run on 24 GB VRAM homelab workstation?

Heavy local models and homelab inference Examples: RTX 3090, RTX 4090.

Preset VRAM24 GB

System memory64 GB

Use caseHeavy local models and homelab inference

Best clean fits

Small multimodal local assistant and low-resource setups

Small local coding assistant and agent tool generation

Fast local chat and simple agent tasks

Fast local chat, lightweight agents, low-cost local testing

Model	Size	Q4 need	Status	Calculator
Gemma 3 4B	4B	3.50 GB	Runs locally	Open calculator
Qwen2.5 Coder 7B	7B	5.50 GB	Runs locally	Open calculator
Mistral 7B	7B	5.50 GB	Runs locally	Open calculator
Llama 3.1 8B Instruct	8B	6.00 GB	Runs locally	Open calculator
Qwen3 8B	8B	6.00 GB	Runs locally	Open calculator
DeepSeek R1 Distill Qwen 8B	8B	6.00 GB	Runs locally	Open calculator
Gemma 3 12B	12B	9.00 GB	Runs locally	Open calculator
Qwen2.5 Coder 14B	14B	10.5 GB	Runs locally	Open calculator
DeepSeek R1 Distill Qwen 14B	14B	10.5 GB	Runs locally	Open calculator
Phi-4 14B	14B	10.5 GB	Runs locally	Open calculator
Gemma 3 27B	27B	18 GB	Runs locally	Open calculator
Qwen2.5 Coder 32B	32B	21 GB	Runs locally	Open calculator
DeepSeek R1 Distill Qwen 32B	32B	21 GB	Runs locally	Open calculator
Mixtral 8x7B	46.7B	28 GB	RAM offload	Open calculator
Llama 3.1 70B Instruct	70B	44 GB	RAM offload	Open calculator