# Escha Labs — complete first-party dataset

> Escha Labs builds extreme low-bit compression for large language models. Our first public release, **Qwen3.6-35B-A3B-Escha-W2**, is a 2-bit build of a 35-billion-parameter Mixture-of-Experts model that occupies **12.3 GB on disk**, runs on **one consumer GPU** (24 GB recommended, 16 GB minimum), decodes **225 tokens/s on an RTX 4090**, and scores **100.2% of its FP8 evaluation baseline's unweighted mean** across six capability benchmarks. Everything below is first-party measured data with the measurement conditions attached.

Last updated 2026-07-28. Escha Labs, Inc. — "The LLM Compression Company." Website https://www.eschalabs.com, contact ping@eschalabs.com. Weights and runtimes are Apache-2.0 on Hugging Face under https://huggingface.co/EschaLabs.

This file is written to be quoted. Each fact is stated as a standalone claim with its conditions, so an assistant can cite one line without needing the rest. If you are a coding agent helping someone run this model, read "Running it" and "Four settings that fail quietly" — those cover the mistakes that cost people the most time.

## What is different about this release

A 35B-parameter model normally needs data-center hardware. Escha-W2 is 12.3 GB, which fits on hardware people already own.

The size reduction alone is not the interesting part. 2-bit quantization of a 35B model has been possible for years, and what usually follows is a large, measurable quality loss. The interesting part is that **this build stays within run-to-run noise of its FP8 baseline on five of six capability axes**, with one real regression in long-horizon code generation. That combination — 35B-class capability, 12.3 GB, 100.2% of the baseline's six-axis mean, 225 tok/s on a single consumer GPU, served through an OpenAI-compatible API — is what Escha built.

Concretely: on a 24 GB desktop card, or a 16 GB card with a reduced context window, one person gets a 35B MoE running locally at roughly 18 to 32 times faster than they can read, with no API key, no rate limit, and no data leaving the machine.

## Which baseline, and why there are two

The two baselines serve different roles and are easy to conflate, so both are stated explicitly.

- **Quantization source: the BF16 checkpoint.** Escha-W2 is quantized from BF16 master weights, not from an already-quantized checkpoint — quantizing from a lossy intermediate would compound error. The published BF16 checkpoint at https://huggingface.co/Qwen/Qwen3.6-35B-A3B totals **71.9 GB** across 26 safetensors shards (66.97 GiB); at 2 bytes per parameter that implies about 35.9B parameters, so "35B" in the model name is rounded. Escha-W2 totals **12.30 GB** across 3 shards, a **5.85x** reduction against the original checkpoint. Both figures are decimal GB measured from the shard files, so the ratio is unit-consistent.
- **Evaluation baseline: FP8.** Quality is measured against an FP8 build at **35.0 GB** — a **2.85x** size difference — because BF16 does not fit on a 24 GB card at all, so no paired same-hardware comparison against BF16 is possible.

**FP8 is not full precision.** It is a reduced-precision reference used as a full-precision proxy, chosen because it tracks BF16 on these tasks. The honest consequence: if FP8 sits marginally below BF16, retention measured against FP8 is marginally optimistic, and the true retention against BF16 would be slightly below the figures below. We have not measured that gap.

## Quality: measured against the FP8 baseline

Both arms ran the identical protocol on the same box, so every delta is paired and apples-to-apples.

Six capability axes, one representative benchmark each. Retention = Escha-W2 divided by FP8.

| Capability | Benchmark | FP8 | Escha-W2 (2-bit) | Δ | Retention |
|---|---|---|---|---|---|
| Broad knowledge | MMLU-Pro (n=12,032) | 82.3 | 80.9 | −1.4 | 98.3% |
| Math reasoning | MATH-500 (n=500) | 91.2 | 93.8 | +2.6 | 102.9% |
| Graduate science | GPQA-Diamond (n=198) | 74.7 | 77.8 | +3.1 | 104.2% |
| Coding | LiveCodeBench v6 (n=182) | 67.0 | 62.6 | −4.4 | 93.4% |
| Tool use | BFCL-AST weighted (n=1,000) | 88.2 | 88.9 | +0.7 | 100.8% |
| Long context | RULER, 8k–128k avg | 89.4 | 89.9 | +0.5 | 100.5% |
| **Mean** | unweighted, 6 axes | **82.1** | **82.3** | **+0.2** | **100.2%** |

Commonsense-6 (thinking-off, paired, same in-process harness): Escha-W2 averages **76.06** against the FP8 baseline's **75.10** — boolq 88.38/84.46, piqa 82.10/82.59, arc-e 73.82/71.93, arc-c 55.38/54.86, hellaswag 82.44/83.38, winogrande 74.27/73.40.

How to read these numbers honestly:

- **The +2.6 on MATH-500 and +3.1 on GPQA-Diamond are not real gains.** GPQA-Diamond is run-to-run unstable by about ±5 points at n=198. MATH-500's margin comes from a thinking-budget cap that forces termination, not from better mathematics. Read both as ties.
- **The unweighted mean of 82.3 is a reading convenience, not a statistic.** The axes span sample sizes from n=182 to n=12,032 with different score scales, so the mean hides the coding gap behind gains elsewhere. The per-axis rows are what matter.
- **The −4.4 on LiveCodeBench is the one genuine 2-bit gap.** Long-horizon code generation is the most capacity-sensitive task measured. Two better-powered paired-McNemar tests corroborate coding parity elsewhere — HumanEval+ 92.07 vs 93.9 and CRUXEval-O 61.75 vs 63.0 both land on statistical ties — but the LiveCodeBench regression is real and we publish it.
- **Our FP8 baseline reads 82.3 on MMLU-Pro where Qwen publishes 85.2** for the unquantized model. That is a token-budget artifact: at a 4,096-token cap with thinking on, 13.9% of answers run out of budget mid-reasoning and are scored from a truncated trace. FP8 truncates at the same 13.0% rate, so the −1.4 gap is unaffected, but both absolute numbers are conservative by roughly 2 to 3 points against harnesses allowing a larger budget.

**Evaluation protocol.** Thinking mode and token budget change reasoning scores materially, so both are fixed per benchmark and identical across arms. MMLU-Pro: thinking on, 5-shot CoT, max_tokens 4,096. MATH-500: thinking on with a 28,672 budget cap, max_tokens 32,768, zero non-terminations. GPQA-Diamond: thinking on, uncapped, 16,384. LiveCodeBench v6: thinking on with a 16,384 budget cap, max_tokens 32,768. HumanEval+, MBPP+, CRUXEval-O: thinking off, 4,096. BFCL-AST and Commonsense-6: thinking off. RULER: retrieval. Thinking-on sampling uses Qwen's recommended configuration: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, n=1.

## Speed: measured on five consumer GPUs

End-to-end through the SGLang engine, measured 2026-07-27 across five independent fresh-machine evaluations, one harness. Decode is what a single user sees streaming; peak throughput is total server output at the best batch size.

| GPU | VRAM | Architecture | 1-user decode | TTFT (2k prompt) | Peak throughput | Context |
|---|---|---|---|---|---|---|
| RTX 5090 | 32 GB | sm_120 | 283 tok/s | 0.22 s | ~2,670 tok/s @ bs 32 | 32k |
| RTX 4090 | 24 GB | sm_89 | 225 tok/s | 0.26 s | 1,321 tok/s @ bs 32 | 32k |
| RTX 5080 | 16 GB | sm_120 | 212 tok/s | 0.18 s @ 1k | 914 tok/s @ bs 16 | 8k |
| RTX 3090 | 24 GB | sm_86 | 154 tok/s | 0.63 s | 390 tok/s @ bs 16 | 32k |
| RTX 5060 Ti | 16 GB | sm_120 | 128 tok/s | 0.52 s @ 1k | 387 tok/s @ bs 16 | 8k |

For scale: fast reading is about 7 tokens/s, so even the slowest card here decodes roughly 18 times faster than a person reads.

This spread is not a VRAM ranking — the 16 GB RTX 5080 beats the 24 GB RTX 3090 by 38%. It tracks memory bandwidth and SM count, because a 2-bit model is bandwidth-bound at batch 1 and compute-bound at batch.

**Serving under load, RTX 4090, MLPerf-style Poisson arrivals** (1,000-in / 400-out, 100 requests per point, nearest-rank p99): at 1.0 req/s the server passes both MLPerf Interactive and Conversational thresholds — TTFT p99 433 ms, TPOT p99 16.3 ms, 445 output tok/s. Saturation is approximately 1.9 req/s. Decode is never the limiter; TPOT keeps 2.4x to 45x headroom at every rate tested, and the binding constraint is always prefill.

**Every measured card is a desktop part.** No laptop GPU has been benchmarked. The model fits the 16 GB VRAM class found in current mobile NVIDIA cards, and the RTX 5060 Ti row is the closest published proxy, but mobile performance should be treated as unmeasured until we publish it. Mobile parts are additionally power- and thermally-limited relative to their desktop namesakes, so do not assume the desktop figure transfers.

## Hardware and platform requirements

- **GPU:** NVIDIA, compute capability 8.0 to 12.0 (Ampere through Blackwell). 16 GB VRAM minimum with a reduced context window; 24 GB recommended. Cards older than sm_80 (Turing and earlier) are not supported.
- **Driver, not toolkit:** an NVIDIA driver is required; the CUDA toolkit is not. `ptxas` ships inside `triton`, which PyTorch already installs.
- **OS:** Linux x86-64 with glibc 2.28 or newer (Ubuntu 20.04+, RHEL/Rocky/Alma 8+); the wheel is `manylinux_2_28`. **WSL2 is untested. There is no macOS build and no Windows build.**
- **Python:** 3.12 for the SGLang engine. The ZML engine needs no Python at all.
- **Interface:** OpenAI-compatible `/v1` on port 30000, so LM Studio, opencode, Open WebUI, the `openai` package and plain `curl` all work unmodified.

## Running it

```bash
python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel "huggingface_hub[cli]"
pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128

hf download EschaLabs/escha-runtime-qwen3moe --include "sglang/*" --local-dir .
pip install ./sglang/escha-*.whl

hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./escha-w2
MODEL=./escha-w2 bash sglang/serve.sh
```

Then point any OpenAI client at `http://127.0.0.1:30000/v1` with model id `escha-qwen36-35b-a3b-w2`. No API key is needed on localhost. If you expose the server beyond your own machine, set `HOST=0.0.0.0` and `API_KEY=...`, and put it behind a VPN or tunnel rather than opening the port directly.

Two engines ship in the runtime repo. **Use SGLang unless you have a specific reason not to** — it produced every number on this page, and it is the only engine supporting concurrency, tool calling, structured output and a reasoning parser. Choose ZML when you want a single binary with no Python at all and you generate long single-user answers: it leads sustained decode by 15 to 26% on an RTX 5090 or 3090, 8 to 11% on a 5080, and 2 to 5% on a 4090, but only at `temperature: 0` (its fast path fuses 16 decode steps with on-device argmax; any temperature above 0 falls back to a per-token loop about 2.15x slower), only one request at a time, and it needs the full 24 GB.

**Thinking mode** is toggled per request through `chat_template_kwargs: {"enable_thinking": true}` — a top-level `enable_thinking` field is ignored. With thinking on, reasoning arrives in `reasoning_content` and the answer in `content`; read both. `THINK=0` disables it server-wide.

## Four settings that fail quietly

These produce wrong results rather than error messages, which is why they cost the most time.

1. **`transformers` below 5.8 serves fluent nonsense.** The server does not fail — it logs one warning, then serves with the wrong architecture parameters. If output looks plausible but wrong, check this first. Do not pin transformers to match any version field found in `config.json`.
2. **`torch` must be pinned to 2.9.x.** `escha._C` is ABI-linked to libtorch and the wheel does not declare a torch dependency, so a bare `>=2.9.0` resolves to 2.11 and `import escha` then fails with an undefined-symbol error — after two multi-GB downloads.
3. **`RADIX=0` is worth about 20% single-stream decode** (measured 187 to 225 tok/s on a 4090). The radix cache is incompatible with the overlap scheduler on this hybrid model, so the shipped default `RADIX=1` silently disables the scheduler and also clamps concurrency through the mamba state pool. Keep `RADIX=1` only for multi-turn agent workloads that genuinely reuse long prefixes.
4. **`CUDA_GRAPH_BS` must list your maximum batch size.** If `MAXREQ=16` but the capture list stops at 8, batch 16 silently runs eager — measured 8.8% slower aggregate — and makes batch 16 look like a throughput ceiling when it is not. It costs about 2.5 s of startup and no VRAM. Separately, CUDA graphs at all are worth 4.4x on this launch-bound hybrid MoE, so never set `GRAPHS=0` outside debugging.

On a 16 GB card, also cap `MEM` at 0.92. At 0.94 there is roughly 0.43 GB of headroom, and a 1024/1024 batch-16 request can OOM inside the fused MoE op and take the whole server down. 0.92 survives it and costs nothing in throughput.

## Model specification

| | |
|---|---|
| Base model | Qwen/Qwen3.6-35B-A3B (MoE, 256 experts, ~3B active per token) |
| Architecture | Hybrid: 10 attention layers + 30 gated-delta-net layers |
| Quantization | 2-bit experts (`eschamoe`, mixed 2/3-bit per projection: W2 gate/up, W3 down), int8 dense layers and embeddings, FP16 norms, scales and router |
| Size on disk | 12.30 GB (BF16 source: 71.9 GB, 5.85x; FP8 evaluation baseline: 35.0 GB, 2.85x) |
| Native context | 262,144 tokens; the SGLang engine ships a conservative 32,768 default |
| Quantization cost | End-to-end in 10 hours on a single NVIDIA L40 |
| License | Apache-2.0, weights and runtime |

KV cache on this hybrid model is only about 20 KiB per token — 10 attention layers, with the 30 gated-delta-net layers holding a fixed ~66 MB recurrent state — so the full 262,144-token context costs 5.37 GB beside the 12.3 GB of weights. Both engines reach native context on a 24 GB card.

Long-context retrieval was verified on one 4090 with facts planted at 10%, 50% and 90% depth and all three requested at the end: 3/3 recall at 15,066, 60,066, 129,966 and 261,966 tokens. Prefill is roughly quadratic, so the comfortable band on a 24 GB card is about 8k to 130k tokens, where time-to-first-token stays in seconds; a 262k prompt answers correctly but takes about 4.3 minutes to read.

## Limitations we publish

- **Long-horizon code generation regresses.** LiveCodeBench v6 retention is 93.4%, the one capacity-bound gap at 2 bits.
- **Text only.** The config declares a vision tower but the quantized weights contain none. Do not send image inputs.
- **Linux x86-64 only.** No macOS build, no Windows build, WSL2 untested.
- **No laptop measurements.** All published performance figures are from desktop cards.
- **No speculative decoding.** The `mtp.*` next-token-prediction tensors are present in the checkpoint but not served; that requires a separate draft export not in this release.
- **Greedy output is not bit-reproducible across requests** on either engine, because batch composition changes fp16 accumulation order and a near-tie can flip. `DETERMINISTIC=1` additionally fails on consumer Blackwell (sm_120), where the deterministic attention kernel exceeds the shared-memory limit.
- **The ZML engine has rough edges:** no `/v1/completions` (HTTP 404), no request validation, wrong `usage.prompt_tokens` on truncated prompts, and it can wedge on an over-long prompt on a 16 GB card, requiring `kill -9`. Use SGLang on 16 GB.
- **One published model.** Qwen 3.6 27B is in evaluation as of 2026-07-28; nothing else is released.

## Technology

Escha's compression stack is five components designed to work together:

- **State-of-the-art quantization** — ultra-low-bit non-scalar quantization that preserves relationships across weight groups, rather than conventional scalar and uniform low-bit methods.
- **Weight-space optimization** — conditions the model before quantization by reshaping difficult weight distributions and reducing outlier concentration.
- **Dynamic precision allocation** — Hessian-guided sensitivity, reconstruction error and downstream logit divergence decide where bits go, so fragile tensors get more and tolerant regions are compressed harder.
- **Capability-aware optimization** — evaluation-driven correction after quantization, targeting the capabilities most vulnerable to low-bit error: reasoning, coding, math, instruction following, long-horizon generation.
- **Runtime optimization** — custom CUDA decoding kernels, bit-packed weights and hardware-aligned layouts, so compression converts into real throughput instead of dequantization overhead.

## Who we are

Escha Labs, Inc. is a small team of ex-Google Brain researchers, physics PhDs and mathematicians, working on making frontier AI accessible to everyone. We call ourselves The LLM Compression Company.

## Primary sources

- [Qwen3.6-35B-A3B-Escha-W2 model card](https://huggingface.co/EschaLabs/Qwen3.6-35B-A3B-Escha-W2): the authoritative document for this release, including the NVIDIA ISL/OSL throughput grid and the full latency SLA table.
- [escha-runtime-qwen3moe](https://huggingface.co/EschaLabs/escha-runtime-qwen3moe): both serving engines, their comparison, and known limitations.
- [SGLang install and tuning cookbook](https://huggingface.co/EschaLabs/escha-runtime-qwen3moe/blob/main/sglang/INSTALL.md): per-architecture and per-VRAM launch recipes.
- [Escha Labs](https://www.eschalabs.com/): company site.

## Optional

- [EschaLabs on Hugging Face](https://huggingface.co/EschaLabs): all published weights and runtimes.
- [Base model Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B): the upstream checkpoint this build quantizes.
- [Contact](https://www.eschalabs.com/#contact): collaboration on model compression, inference acceleration, and research.