Benchmarks

Which model can do the work?

Kitlig users test local and hosted models on Kitlig's own jobs: calling the Assistant's tools, writing an edit plan and a Lottie animation, translating dialogue and describing a picture. Every row is a model on a real machine.

Assistant

Chat, agents and tool calls

Local

Qwen3.6-35B-A3B-MTP-GGUF

NVIDIA GeForce RTX 3090 24 GB · 136 tok/s

Hosted

deepseek-v4.1-flash

OpenRouter · 152 tok/s

Cheapest API that passes

claude-haiku-5.5

OpenRouter · <$0.01 a run

Planning

Edit plans and Lottie animations

Local

Qwen3.6-35B-A3B-MTP-GGUF

NVIDIA GeForce RTX 3090 24 GB · 136 tok/s

Hosted

deepseek-v4.1-flash

OpenRouter · 152 tok/s

Cheapest API that passes

claude-haiku-5.5

OpenRouter · <$0.01 a run

Describe & Tag

Looking at pictures

Local

Qwen3.6-35B-A3B-MTP-GGUF

NVIDIA GeForce RTX 3090 24 GB · 136 tok/s

Hosted

deepseek-v4.1-flash

OpenRouter · 152 tok/s

Cheapest API that passes

llama-4-maverick

OpenRouter · <$0.01 a run

Translation

Dubbing translations

Local

Qwen3.6-35B-A3B-MTP-GGUF

NVIDIA GeForce RTX 3090 24 GB · 136 tok/s

Hosted

deepseek-v4.1-flash

OpenRouter · 152 tok/s

Cheapest API that passes

llama-4-maverick

OpenRouter · <$0.01 a run

Must pass

49 of 49 model and machine combinations · 61 shared runs

	Runs on
deepseek/deepseek-v4.1-flash	OpenRouter	100	152	✓	✓	✓	✓	✓	$0.011 $0.05 / $0.20 per 1M	2
anthropic/claude-haiku-5.5	OpenRouter	100	108	✓	✓	✓	✓	✓	<$0.01 $0.10 / $0.50 per 1M	2
google/gemini-3.6-flash	OpenRouter	100	104	✓	✓	✓	✓	✓	$0.078 $0.75 / $3.8 per 1M	1
mistralai/mistral-large-4-0	OpenRouter	100	86	✓	✓	✓	✓	✓	$0.057 $0.70 / $2.1 per 1M	2
anthropic/claude-sonnet-5.5	OpenRouter	100	74	✓	✓	✓	✓	✓	$0.181 $2.0 / $10 per 1M	2
google/gemini-3.8-flash	OpenRouter	100	69	✓	✓	✓	✓	✓	$0.067 $0.83 / $4.1 per 1M	2
deepseek/deepseek-v4-pro-0813	OpenRouter	80	69	✓	✓	✓	✓	✗	$0.024 $0.11 / $4.4 per 1M	2
anthropic/claude-opus-5.5	OpenRouter	100	67	✓	✓	✓	✓	✓	$0.428 $4.0 / $20 per 1M	2
google/gemini-3.1-pro-preview	OpenRouter	100	60	✓	✓	✓	✓	✓	$0.309 $2.0 / $12 per 1M	1
claude-haiku-5-5	Claude Code	100	58	✓	✓	✓	✓	✓	subscription <$0.01 a run via API	1
openai/gpt-6.1-terra-pro	OpenRouter	100	51	✓	✓	✓	✓	✓	$0.047 $1.0 / $5.0 per 1M	1
claude-sonnet-5-5	Claude Code	100	46	✓	✓	✓	✓	✓	subscription $0.181 a run via API	1
openai/gpt-6.1-terra	OpenRouter	100	41	✓	✓	✓	✓	✓	$0.015 $1.0 / $5.0 per 1M	1
anthropic/claude-fable-5.1	OpenRouter	100	40	✓	✓	✓	✓	✓	$0.857 $10 / $50 per 1M	1
moonshotai/kimi-k3.1-pro	OpenRouter	100	38	✓	✓	✓	✓	✓	$0.139 $2.0 / $10 per 1M	2
x-ai/grok-4.7	OpenRouter	100	35	✓	✓	✓	✓	✓	$0.187 $2.0 / $6.0 per 1M	2
claude-opus-5-5	Claude Code	100	35	✓	✓	✓	✓	✓	subscription $0.428 a run via API	1
unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-IQ4_NL	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	100	136	✓	✓	✓	✓	✓	free · your hardware	1
unsloth/gemma-4-31B-it-qat-GGUF UD-Q4_K_XL	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	100	69	✓	✓	✓	✓	✓	free · your hardware	1
unsloth/Muse-Glimmer-30B-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	100	56	✓	✓	✓	✓	✓	free · your hardware	1
gemini-3.6-flash	Gemini CLI	100	27	✓	✓	✓	✓	✓	subscription $0.078 a run via API	1
prism-ml/Ternary-Bonsai-27B-gguf	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	100	64	✓	✓	✓	✓	✓	free · your hardware	1
gpt-6.1-terra	Codex	100	27	✓	✓	✓	✓	✓	subscription $0.015 a run via API	1
gemma-4-26B-A4B-it-qat-UD-Q4_K_XL	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	80	163	✓	✓	✓	✓	✗	free · your hardware	1
claude-fable-5-1	Claude Code	100	26	✓	✓	✓	✓	✓	subscription $0.857 a run via API	1
google/gemini-4-argon	OpenRouter	100	25	✓	✓	✓	✓	✓	$0.110 $2.0 / $10 per 1M	2
unsloth/gpt-oss-20b-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	80	153	✓	✓	✓	✓	✗	free · your hardware	1
openai/gpt-6.1-sol	OpenRouter	100	22	✓	✓	✓	✓	✓	$0.030 $2.0 / $10 per 1M	1
unsloth/Qwen3.8-27B-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	65	30	✓	✓	✗	–	✓	free · your hardware	1
gemini-3.1-pro-preview	Gemini CLI	100	16	✓	✓	✓	✓	✓	subscription $0.309 a run via API	1
openai/gpt-6-astra	OpenRouter	100	16	✓	✓	✓	✓	✓	$0.273 $10 / $50 per 1M	1
auto	Gemini CLI	100	16	✓	✓	✓	✓	✓	subscription	1
gpt-6-astra	Codex	100	15	✓	✓	✓	✓	✓	subscription $0.273 a run via API	1
mudler/KAT-Coder-V2.5-Dev-APEX-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	60	111	✓	✓	✗	✓	✗	free · your hardware	1
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	45	12	✓	✓	✗	–	✗	free · your hardware	1
unsloth/gemma-4-E2B-it-qat-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	100	298	✓	✓	✓	✓	✓	free · your hardware	1
Qwen3.6-27B-Q4_K_M	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	45	7	✓	✓	✗	✗	✗	free · your hardware	1
DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	80	7	✓	✓	✗	✓	✓	free · your hardware	1
empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	80	11	✓	✓	✗	✓	✓	free · your hardware	1
gpt-6.1-sol	Codex	85	18	✓	✗	✓	✓	✓	subscription $0.030 a run via API	1
z-ai/glm-5.3	OpenRouter	65	178	75%	50%	✓	✓	✗	$0.013 $0.08 / $3.6 per 1M	2
unsloth/LFM2.5-8B-A1B-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	59	291	80%	✗	✓	✓	✗	free · your hardware	1
meta-llama/llama-4-maverick	OpenRouter	81	13	63%	50%	✓	✓	✓	<$0.01 $0.35 / $1.0 per 1M	2
yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	42	57	40%	✓	✗	✓	✗	free · your hardware	1
OBLITERATUS/Qwen3.8-27B-OBLITERATED	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	70	51	none	✓	✓	✓	✓	free · your hardware	1
prism-ml/Bonsai-27B-gguf	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	0	Did not load on this machine							1
unsloth/diffusiongemma-26B-A4B-it-GGUF	NVIDIA GeForce RTX 3090 24 GB AMD Ryzen 5 5500 · 32 GB RAM	0	Did not load on this machine							1
qwen/qwen3.8-flash	OpenRouter	0	Did not load on this machine							1
qwen/qwen3.8-max-0902	OpenRouter	0	Did not load on this machine							1

How to add yours. In Kitlig, open Settings ▸ Model Benchmarks, test a model on your own server, through your OpenRouter or OpenAI key, or on your Claude Code, Codex or Gemini CLI subscription, then press Share. Only activated copies of Kitlig can share, and we never publish who shared a result, your server's address or your keys.

Reading the numbers. Speed is the median tokens per second across runs, so it depends on the machine, the quantisation and whether the model thinks first. Hosted speeds include the network. A tick means every run passed; a percentage means some runs failed. Cost is what one full test run cost on the provider's bill (about a dozen calls), or an estimate from OpenRouter's list price when it wasn't reported. Local models cost nothing per call, and the CLI agents run on a subscription.