Research

Local LLM face-off: 19 models, one benchmark

A budget research with LM Studio: 6 deterministic tests, 19 models that ran on a 16 GB RAM machine (13.2 available), and a two-axis rubric (content × format) that changes the ranking completely.

2026-09-13

40 min

Choosing a local language model used to boil down to a single question: which one is the best? After running 20 models through LM Studio on my own machine, I can say that question is framed wrong.

The right question has three parts: what fits in 16 GB of RAM (in practice, 13.2 GB available), what you consider a correct answer, and how long you are willing to wait. At its core this comparison is a budget research: figuring out what can realistically run on a regular computer — in my case one with 16 GB of RAM (13.2 available), loaned by the government for a consultancy with intensive AI use — rather than on a GPU rack.

The uncomfortable twist: most models knew the long-context answer but wrote it without the required format. Here format is part of the test, so that answer is worth half, not zero and not everything. Once you separate "knows the answer" from "answers in the required format," the ranking changes entirely.

TL;DR

This is a budget research: figuring out which LLM can run on a regular computer. We tested 1.2B–27B models with the same bench of 6 deterministic tests via LM Studio, on the same machine —16 GB of RAM (13.2 available), loaned for a consultancy with intensive AI use— and the same script. Each answer is scored on two axes: correct content and the format the test demands. Delivered exactly as asked → 1.0; correct content but unmet format instruction → 0.5; wrong or empty → 0. Under that rule format weighs, but it does not wipe out content. Most models retrieved the long-context needle; only 11 passed the math test; and logic left almost everyone at the door.

Dataset19valid models (1.2B → 27B)

Tests6same deterministic bench

Rubric1 · ½ · 0content × format

Long context84.2%16 of 19 points (weighted)

Couldn't run1over the 13.2 GB available

Methodology

To keep the comparison fair, only one variable changed: the model. Everything else —machine, prompt, temperature, scoring rule— was fixed and versioned before the first test ran.

Method

One machine, one bench, one script

Each model was loaded on its own in LM Studio against the local endpoint http://localhost:1234/v1/chat/completions. Temperature fixed at 0.1 and a generous 2048-token cap per answer (just a ceiling: no saved answer in the dataset was truncated), so the only variable between models was the model itself. Each test ran once, and we saved the raw answer, the script’s strict verdict, and the latency. Nothing was re-run to "improve" results: the content-vs-format axis was computed afterwards, on the saved answers.

Budget13.2 GBcan the model fit with context?

Load1×1one model at a time in LM Studio

Run6deterministic tests (temp 0.1)

Log.jsonraw answer + latency per test

Rubric1 · ½ · 0content (re-checked) × format (strict)

The 6 tests

IF-01 · IFStrict JSONProduce a valid JSON with exactly 3 recommendations.

IF-02 · IFNo letter "e"Same task, no "e" or "é" allowed: a near-impossible ceiling in Spanish.

TA-01 · TAExact extractionExtracts customer, product and amount_usd from a text, as JSON.

LOG-01 · LOGSeating under rules3 people and 3 seats with constraints; answer in \boxed{}.

MATH-01 · MATHFill-drain ratesA tank with inflow and outflow; answer in \boxed{}.

LC-01 · LCLong contextA key hidden in ~6.4k filler tokens; answer in \boxed{}.

The five dimensions

Five different loops, to be precise: each test demands a different kind of reasoning and a different output format.

Instruction following47.4%

Design ceiling: the JSON part everyone passes; the "no e" test is nearly impossible in Spanish.

Text analysis89.5%

Exact extraction: universal, does not set models apart.

Logical reasoning42.1%

The most selective: only Ministral 3 · 8B, Qwen3-VL 8B, Ministral 3 · 14B Reasoning, Qwen3 4B, Bonsai 27B, Phi-4 Mini Reasoning, Liquid LFM 1.2B, and Phi-4 Reasoning-Plus.

Math reasoning60.5%

Math reasoning passed by: Ministral 3 · 8B, Qwen3-VL 8B, Ministral 3 · 14B Reasoning, Qwen3 4B, Bonsai 27B, Granite 4-H Tiny, Nemotron Nano 4B, Qwen3-VL 4B, Phi-4 Mini Reasoning, Phi-4, and Phi-4 Reasoning-Plus.

Long context84.2%

13/19 fulfill the format; the rest return the key content-only (0.5 each).

Reproducibility

This comparison is not a ranking of opinions: it is a versioned pipeline. If you have LM Studio with a loaded model, you can replicate the whole experiment on your machine, model by model, without asking me anything. The prompts are not a secret: they are data.

Pipeline

From model to score, nothing touched by hand

01Loadmodel in LM Studio, local server on :1234, one at a time

02Runevaluar_lmstudio.py: 6 tests, same temperature and max_tokens

03Reportresultados/reporte_*.json: raw answer + verdict + latency

04Scorepuntuacion_final.py: 1·½·0 rubric on saved answers

05Publishdataset regenerated: the post updates with one command

git clone https://github.com/CrAvila/Evaluador_LLM_Studio
cd Evaluador_LLM_Studio

# 1) load the model in LM Studio and start the local server on :1234
# 2) run the suite: 6 deterministic tests, one per model
python evaluar_lmstudio.py

# 3) after each run, regenerate the post dataset
python scripts/puntuacion_final.py --export-json --web <path-to-cravila-web>

Prompts

The exact prompts, unfiltered

Each test was designed before seeing any result and executed with these exact strings. Nothing was tweaked mid-run: if you want to replicate this on your own machine, you do not need to ask for prompts — they are here, exactly as sent to the model.

IF-01Strict JSON[IF]

Genera una lista de exactamente 3 recomendaciones de productividad. Tu respuesta debe ser EXCLUSIVAMENTE un objeto JSON con la clave 'recomendaciones' que contenga una lista de strings. No incluyas texto antes ni después del JSON.

Validation: The script demands valid JSON with "recomendaciones" as a list of exactly 3 strings.

IF-02No letter "e"[IF]

Escribe un párrafo sobre la importancia de la ciberseguridad. RESTRICCIÓN ESTRICTA: La respuesta NO debe contener la letra 'e' (ni mayúscula ni minúscula) bajo ninguna circunstancia.

Validation: The answer cannot contain "e" or "é" and must be longer than 30 characters.

TA-01Exact extraction[TA]

Analiza el siguiente texto y extrae el nombre del cliente, el producto y el monto.
Texto: 'El cliente Juan Pérez realizó una compra de una Laptop Pro por el valor de 1250 USD el día de ayer.'
Responde ÚNICAMENTE en el siguiente formato JSON:
{"cliente": "...", "producto": "...", "monto_usd": 000}

Validation: All three fields must match exactly (customer "Juan Pérez", product "Laptop Pro", amount_usd = 1250).

LOG-01Seating under rules[LOG]

Problema Lógico:
A, B y C están en una fila. A no está al lado de B. C está a la derecha de A.
¿Cuál es el orden exacto de izquierda a derecha?
Opciones:
A) A, C, B
B) B, A, C
C) C, B, A
D) A, B, C

Piensa paso a paso. Al final de tu respuesta, indica la opción correcta dentro del comando \boxed{}, por ejemplo: \boxed{A}.

Validation: The regex extracts \boxed{...} and demands the value equals "A".

MATH-01Fill-drain rates[MATH]

Un estanque se llena con un grifo en 4 horas y se vacía con un desagüe en 6 horas. Si ambos están abiertos al mismo tiempo, ¿cuántas horas tardará en llenarse el estanque?
Resuelve el problema paso a paso. Muestra tu resultado final dentro del comando \boxed{número}.

Validation: The regex demands \boxed{12} (or "12 hours").

LC-01Long context[LC]

A continuación se presenta un registro extenso de datos:
(300 × "Dato de relleno sin importancia. " ≈ 6.400 tokens)
CLAVE_SECRETA_SISTEMA: 'Alfa-9921-Zeta'
(300 × "Dato de relleno sin importancia. ")

¿Cuál es la CLAVE_SECRETA_SISTEMA mencionada en el texto? Responde únicamente con el valor dentro de \boxed{{}}.

Validation: The regex demands \boxed{Alfa-9921-Zeta}.

Rubric

The rule that decides each point

A full point (1.0) is only paid when the answer arrives exactly as asked: correct content and the format the test demands. Deductions come from there: not following the instruction —even when it carries ambiguity— is part of the exercise and of what is expected from AI today: the instruction is followed, not negotiated.

1Delivered exactly as asked: content and format correct

0.5Correct content, but the format instruction was not followed

0Wrong, empty, or malformed answer

Results

Nineteen models ran on the same machine, six tests each. Of 20 candidates, one was left out of the RAM budget (Qwen3.5 9B); the other 19 produced a complete dataset. The final order comes out on its own: weighted average under the rubric, with ties broken by latency.

Results

The full ranking: 19 models, 6 tests each

Click a row to see each test and the answer that model gave.

#	Model	SI	TA	LO	MA	CL	Overall	Latency	Empty
1	Ministral 3 · 8BMistral · 8B	50	100	100	100	100	83.3%	70.8 s	—
2	Qwen3-VL 8BQwen · 8B	50	100	100	100	100	83.3%	83.1 s	—
3	Ministral 3 · 14B ReasoningMistral · 14B	50	100	100	100	100	83.3%	117.1 s	—
4	Qwen3 4BQwen · 4B	50	100	100	100	100	83.3%	129.3 s	—
5	Bonsai 27BPrism ML · 27B	50	100	100	100	100	83.3%	706.4 s	1
6	Granite 4-H TinyIBM · ~4B	50	100	0	100	100	66.7%	14.5 s	—
7	Nemotron Nano 4BNVIDIA · 4B	50	100	0	100	100	66.7%	48.5 s	1
8	Qwen3-VL 4BQwen · 4B	50	100	0	100	100	66.7%	60.4 s	—
9	Phi-4 Mini ReasoningMicrosoft · 3.8B	50	0	100	100	100	66.7%	118.6 s	—
10	Liquid LFM 1.2BLiquid AI · 1.2B	50	100	100	0	50	58.3%	7 s	—
11	Phi-4Microsoft · 14B	50	100	0	100	50	58.3%	182.8 s	—
12	Gemma 4 E2BGoogle · ~3B	50	100	0	0	100	50%	35.4 s	2
13	Gemma 4 E4BGoogle · ~5B	50	100	0	0	100	50%	63.3 s	2
14	GPT-OSS 20BOpenAI · 20B	50	100	0	50	50	50%	114.3 s	—
15	GLM-4.6V FlashZ.AI · ~9B	50	100	0	0	100	50%	122.5 s	2
16	Gemma 4 12B QATGoogle · 15B	50	100	0	0	100	50%	183.5 s	2
17	Ministral 3BMistral · 3B	50	100	0	0	50	41.7%	33.2 s	—
18	DeepSeek R1 · Qwen3 8BDeepSeek · 8B	50	100	0	0	50	41.7%	134.2 s	2
19	Phi-4 Reasoning-PlusMicrosoft · 14.7B	0	0	100	100	50	41.7%	626.8 s	—

OVERALL = average of the 6 tests under the 1·½·0 rubric. Each dimension column adds up its tests (IF contributes 2 of 6). "Empty" = tests with a blank answer, worth 0. Latency = per-test average, in seconds. Amber rows: best score; within a tie, the lower latency wins.