An inference runtime that runs massive open-weight MoE models on consumer GPUs by intelligently offloading experts across VRAM, RAM, and SSD. From Qwen3.5 to Arcee Trinity - 35B to 398B parameters.
Run 398B parameter MoE models on consumer GPUs. Only active experts load into VRAM - cold experts live in RAM and SSD, paged in as routing demands them.
.lmpack Model FormatFile-per-expert packaging enables mmap-based memory management. The OS kernel handles caching automatically - hot experts stay in RAM, cold experts page in from NVMe.
Flash attention, multi-GPU pipeline parallelism, and async I/O. On the 35B, 93% of expert lookups are already in VRAM and need no transfer at all. Output verified against llama.cpp on identical weights. OpenAI-compatible API included.
Fair question. Offloading MoE experts is not new, and you should be suspicious of anyone who says it is. llama.cpp has shipped--cpu-moe for a long time, and expert caching with prefetch was published in 2023. The difference is not that we page experts. It iswhere they live and where they run.
--cpu-moe does not stream anything. It parks the expert weights in system RAM and computes them on the CPU, so it needsas much RAM as the model is large. lean keeps experts on disk as individual files and pages only the active working set onto the GPU.
| System RAM used | Experts compute on | |
|---|---|---|
llama.cpp --cpu-moe | 58 GB + 5 GB swap | CPU |
| lean | 7 GB | GPU |
On that machine llama.cpp can just barely hold the 122B, and pays for it in swap. On a 32 GB machine it cannot load it at all. The gap widens with model size: our largest models are 226 GB and 242 GB, so running them the--cpu-moe way would need 256 GB of RAM. lean runs them on 48 GB of VRAM and an NVMe drive.
We are not claiming to be faster. On hardware that can fit the model in RAM, throughput is comparable and llama.cpp is free. The claim is narrower and more useful:model size stops being bounded by your system RAM.
$ curl -sSf https://leanmodels.ai/install.sh | sh
$ lean pull lean-agent-light
$ lean run lean-agent-lightSingle binary. No Python, no Docker, no cloud dependency. Linux and Windows, NVIDIA GPU required.
Three-tier memory hierarchy: VRAM → RAM → NVMe SSD
| Tier | VRAM | RAM | NVMe | Models |
|---|---|---|---|---|
| Minimal | 12 GB | 16 GB (32 GB for lean-coder-welter) | 1.8 TB | lean-coder-welter, lean-agent-light |
| Prosumer | 24 GB | 32 GB (64 GB recommended) | 1.8 TB | lean-agent-middle |
| Enthusiast | 48 GB | 64 GB (128 GB recommended) | 1.8 TB | lean-think-heavy, lean-reason-heavy (coming soon) |
Many open-source MoE models rival proprietary ones but only activate a fraction of their parameters per token. A 242 GB model needs 48 GB of VRAM, not 242 GB. The barrier is fitting them in memory - that's an engineering problem, and we solve it.