Benchmarks

Output quality and offloading performance, measured on real hardware.

Academic Quality Benchmarks

Standard evals to confirm no capability regression from the lmpack pipeline. Each eval shows lean-engine next to the llama.cpp reference on identical weights - each model's own source GGUF, which is not the same quant for every model - close columns mean the offloading engine reproduces the model faithfully.

Model	GSM8Kn=1319 - exact match, strict		HumanEvaln=164 - pass@1		IFEvaln=541 - prompt-level strict
Model	lean	llama.cpp	lean	llama.cpp	lean	llama.cpp
lean-agent-middle (122B)	83.8	83.9	84.1	83.5	49.4	47.3
lean-coder-welter (80B)	91.4	91.3	68.9	70.7	53.6	56.7
lean-agent-light (35B)	85.8	86.4	74.4	75.6	42.1	43.3

Scores are accuracy percentages. Standard lm-evaluation-harness tasks run through lean-engine and the llama.cpp reference on identical weights - each model's own source GGUF, greedy decoding, same input tokens. Quantization differs between models, so compare the two columns within a row rather than across rows. The point is not a leaderboard - it is to confirm the lmpack offloading pipeline introduces no capability regression versus the reference engine.

Offloading Performance

Token generation with full expert offloading, on a single RTX 3090 (24 GB) and on 2x RTX 3090 (48 GB) - 64 GB RAM, NVMe SSD.

Model	Hardware	Prefill	Decode
lean-agent-middleverified	1x RTX 3090 (24 GB)	6-12 tok/s	5-6 tok/s
lean-agent-middleverified	2x RTX 3090	4-8 tok/s	6-8 tok/s
lean-coder-welterverified	1x RTX 3090 (24 GB)	14-30 tok/s	18-20 tok/s
lean-coder-welterverified	2x RTX 3090	9-20 tok/s	18-28 tok/s
lean-agent-lightverified	1x RTX 3090 (24 GB)	16-41 tok/s	30-40 tok/s
lean-agent-lightverified	2x RTX 3090	11-27 tok/s	24-37 tok/s
lean-agent-cruiser	Coming soon - not yet benchmarked on our hardware
lean-think-heavy	Coming later - not yet benchmarked on our hardware
lean-reason-heavy	Coming later - not yet benchmarked on our hardware

Ranges span decode at 64 and 256 generated tokens (prefill at short prompts, cold cache). Single-GPU uses the x16 card; 2-GPU uses pipeline parallelism. Smaller models run fastest on a single card - the engine defaults to single-GPU when the model fits - while the 122B is faster across two. Measured on trunk, 2026-07.

Verified against llama.cpp

Real greedy outputs from lean-engine vs the llama.cpp reference on identical GGUF weights and identical input tokens. Bit-for-bit is not expected: MoE floating-point reduction order differs, so output shares a prefix then diverges into equally valid text.

lean-agent-middlemath

Identical output

Prompt: If a train travels 60 miles in 1 hour, how far does it travel in 2.5 hours? Show your work briefly.

lean-engine

Thinking Process:

1.  **Analyze the Request:**
    *   Given: A train travels 60 miles in 1 hour.
    *   Question: How far does it travel in 2.5 hours?
    *   Constraint: Show work briefly.

2.  **Identify the Rate:**
    *   Speed = Distance / Time
    *   Speed = 60 miles / 1 hour = 60 miles per hour (mph).

3.  **Calculate the New Distance:**
    *   Formula: Distance = Speed × Time
    *   Speed

llama.cpp reference

Thinking Process:

1.  **Analyze the Request:**
    *   Given: A train travels 60 miles in 1 hour.
    *   Question: How far does it travel in 2.5 hours?
    *   Constraint: Show work briefly.

2.  **Identify the Rate:**
    *   Speed = Distance / Time
    *   Speed = 60 miles / 1 hour = 60 miles per hour (mph).

3.  **Calculate the New Distance:**
    *   Formula: Distance = Speed × Time
    *   Speed

Identical to the llama.cpp reference, token for token.

lean-coder-weltercode

Matches for 82 tokens

Prompt: Write a Python function called `is_prime` that checks if a number is prime. Just the code, no explanation.

lean-engine

```python
def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    i = 3
    while i * i <= n:
        if n % i == 0:
            return False
        i += 2
    return True
```

llama.cpp reference

```python
def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False
    i = 3
    while i * i <= n:
        if n % i == 0:
            return False
        i += 2
    return True
```

Tracks the llama.cpp reference for 82 tokens, then makes an equally valid choice.

lean-agent-lightlanguage

Matches for 41 tokens

Prompt: Translate 'Hello, how are you?' to French.

lean-engine

Thinking Process:

1.  **Analyze the Request:**
    *   Source text: "Hello, how are you?"
    *   Target language: French
    *   Task: Translation

2.  **Identify Key Components:**
    *   "Hello": A greeting. Common French equivalents: "Bonjour", "Salut", "Bonjour".
    *   "how are you?": Asking about well-being. Common French equivalents: "comment allez-vous?", "comment vas-tu?", "ça va?", "comment ça va?".

3.  **Determine Register/Context:**
    *   The English phrase "Hello, how are you?" is neutral to slightly formal/standard.
    *   "Hello" -> "Bonjour" (

llama.cpp reference

Thinking Process:

1.  **Analyze the Request:**
    *   Source text: "Hello, how are you?"
    *   Target language: French
    *   Task: Translate the source text to the target language.

2.  **Determine the appropriate French translation:**
    *   "Hello" can be translated as "Bonjour" (standard, daytime), "Salut" (informal), or "Bonsoir" (evening). "Bonjour" is the safest and most common default.
    *   "How are you?" can be translated as "Comment allez-vous?" (formal/plural), "Comment vas-tu?" (informal singular), or "Ça va?" (informal/neutral).
    *

Tracks the llama.cpp reference for 41 tokens, then makes an equally valid choice.

Engine Features

Performance infrastructure built into the runtime. The two hit rates below measure different things: the cache hit rate is how often a needed expert is already in VRAM, while the prefetch hit rate is how accurate the predictor is when it fetches experts ahead of use, which is what absorbs most of the remaining misses.

Verified

Output vs llama.cpp

Same-weights greedy output tracks the reference token-for-token, then diverges only into equally valid text. Checked across 3 models.

93%

VRAM cache hit rate (35B)

Share of expert lookups already resident in VRAM, needing no transfer at all. LRU cache with profile-guided preloading.

66-74%

Speculative prefetch accuracy

Of the experts the router predicts and fetches ahead of the layer that needs them, the share it gets right. Applies to the lookups the cache misses.

1-6 GB/s

Expert preload throughput

Async I/O via background thread pool

2-5s

Model load time

Core weights into VRAM, experts lazy-loaded via mmap

Multi-GPU

Pipeline parallelism

Layers split across GPUs, output bit-identical to single-GPU

Reference Hardware

Every performance number above was measured on this rig. No cloud GPUs. Results are reproducible.

GPU

2x RTX 3090 (24 GB)

CPU

Intel i9-10900

RAM

64 GB DDR4

Storage

PCIe gen3 NVMe

Reference tiers

Tier	VRAM	RAM	Models
Minimal	12 GB	16-32 GB	lean-coder-welter, lean-agent-light
Prosumer	24 GB	32-64 GB	lean-agent-middle
Enthusiast	48 GB	64-128 GB	lean-think-heavy, lean-reason-heavy (coming soon)

Downloads

Each model is a single .lmpack, pulled with one command. The quantization it is built from varies by model and is listed below.

Model	Quantization	Download
lean-think-heavycoming later	Q4_K_M	~241.85 GB (target)
lean-reason-heavycoming later	UD-Q4_K_XL	226.17 GB
lean-agent-cruisercoming soon	UD-Q4_K_XL	155.14 GB
lean-agent-middle	Q4_K_M	77.67 GB
lean-coder-welter	UD-Q4_K_XL	49.72 GB
lean-agent-light	Q4_K_M	22.06 GB

Methodology

Academic benchmarks (GSM8K, HumanEval, IFEval) run vialm-evaluation-harnessagainst the lean serveOpenAI-compatible API, with the llama.cpp reference scored the same way on the same weights. Every score in the table comes from a completed run on the reference rig. Only real, verified numbers are published here.

Offloading benchmarks measure tok/s, VRAM cache hit rate, prefetch hit rate, and expert preload throughput. All numbers come fromlean bench on the reference rig above.

Verification against llama.cpp runs the same greedy prompts through lean-engine and llama.cpp on the same GGUF weights, feeding llama.cpp the exact input token IDs so tokenization is not a variable. Bit-for-bit identity is not the goal: MoE floating-point reduction order differs between engines, so greedy output shares a prefix then diverges into equally valid text. Across the three available models, output tracks the reference closely - the 122B produces several full responses identical token-for-token, the 80B agrees on ~94% of tokens, and every divergence lands on an equivalent word choice, not a worse answer.