Pure C · on-device inference engine

cwick.

Run the whole model, not a smaller one.

A pure-C on-device inference engine running complete, never-pruned GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, MiniMax-M3.1-Flash and DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic checkpoints — 18–475 GiB of weights — on one 128 GiB Apple Silicon Mac, with OpenAI-, Anthropic- and Systemone-compatible APIs.

View on GitHub ↗ Quick start →

C11 · MIT · Apple Silicon · :8001

128 GiB machine

80% RAM ceiling

Why the nameA wick lets a large body of wax burn: 233–475 GiB of weights sit on the SSD, and only a few gigabytes cross the bus for each token.

4

checkpoints supported end to end

233–475 GiB

of weights streamed from the SSD

80%

RAM ceiling, wired, swap stays flat

500K

maximum context

3

API dialects: chat · responses · messages

Under the hood

How a request flows

One server, one resident model, four checkpoint-specific backends. Generation is single-slot; everything else is served concurrently so a long prefill stays watchable.

Client — curl, coding agent, desktop chat app, built-in console

HTTP :8001 · server.csingle-slot generation queue, arrival order

GLM-5.3-Flashspine.c · attention.c
moe.c · pfx.c

DeepSeek-V4.1ds_model.c · ds_ops.c
ds_quant.c

MiniMax-M3.1mm_model.c
mm_prompt.c

DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-hereticq35_model.c
q35_prompt.c · q35_gpu.m

/v1/systemone → jev.c · one prefill, read the distribution

/v1/metrics · /v1/models · /v1/config → answered while the slot is busy

/v1/hub/* → hub.c · download jobs and the model catalog

/ → embedded console, two H5 apps, local models

State on this machine

Residentbf16 spine + KV cache + indexer pools, wired

Expertspin cache in shared memory + direct SSD reads

Snapshotspfx.c checkpoints under --prefix-cache-dir

Computepool.c (NEON threads) or metal.c (zero-copy arena)

C11 plus three Objective-C files — two Metal bridges (metal_objc.m, q35_gpu.m) and a QR encoder. It links only macOS system frameworks — Accelerate, Metal, CoreImage, ImageIO, CoreGraphics — and the system sqlite3. No Python, no BLAS, no runtime conversion step. One binary, four subcommands: setup, run, serve, doctor.

Interfaces

API surface

A short form of the server reference — auth, error shapes and the full endpoint list live in docs/SERVER.md.

OpenAI

OpenAI-compatible

Three generation dialects with streaming, tool calls and the same engine behind them; thinking is off by default unless the request sets it.

/v1/chat/completions · /v1/responses · /v1/completions · /v1/models

Anthropic

Anthropic-compatible

Messages dialect: a top-level system field, content blocks and message_* stream events, including tool use.

/v1/messages

Systemone

Systemone decisions

One prefill reads the answer-position distribution and returns choice, score or noul probabilities — typed decisions without retraining (requires --jev).

/v1/systemone

GET /v1/models

resident checkpoint and what is under ./models, with supported flags

POST /v1/models/switch

validate first, then unload one checkpoint and map another; 409 while generating

POST /v1/completions

raw text completion, streaming or not

POST /v1/chat/completions

messages, stream, tools and reasoning_effort

POST /v1/responses

Responses dialect: input items in, response.* events out

POST /v1/messages

Anthropic dialect: system + content blocks, message_* stream

POST /v1/systemone

typed decisions from one prefill (requires --jev)

GET /v1/metrics

vLLM-compatible names plus five host VM gauges; also /metrics and /health

GET /v1/hub/models

downloaded checkpoints with size and completeness

GET /v1/logs/stream

follow this process's stdout and stderr

POST /v1/auth/login

password → Bearer token and HttpOnly session cookie

PUT /v1/apps/<id>/state

per-app sqlite storage that survives a cleared browser

Generation is single-slot and requests queue in arrival order. Metrics, the model library, the log stream and the console stay answerable while a long prefill runs — that is what makes watching it possible at all.

Checkpoints

Four models, each with its own backend

Not a closed list by design — but every checkpoint needs its own loader, quantizer, prompt renderer and kernels. Decode figures are measured on an M3 Max with 128 GiB and the Metal backend.

Model	On disk	Shape	Context	Decode
GLM-5.3-Flash≈320B parameters · the original model	≈306 GiB62 shards	45 layers · 288 experts · top-8 · FP8-E4M3	500K	3.6 tok/s resident
DeepSeek-V4.1-Flash-1017largest one that still fits a laptop	≈475 GiB48 shards	40 layers · 384 experts · top-6 · MLA + FP4	500K	2.8 tok/s
MiniMax-M3.1-Flashsmallest footprint of the three MoEs	≈233 GiB48 shards	60 layers (3 dense) · 128 experts · top-4 · NVFP4	500K	1.48 tok/s
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-hereticdense, held wholly in RAM	≈18 GiB4 shards	32 layers: 24 Gated DeltaNet + 8 GQA · dense BF16	500K	20.4 tok/s on the GPU

Context is not free: at 500K positions the KV cache, the resident spine and the expert bank all draw from the same 80% RAM ceiling. Every figure on this page is measured, not assumed.

Get it running

Three steps to a local API

No Python, no conversion, no package manager. The commands below are the same ones the README uses.

1 · Clone and build

macOS on Apple Silicon and clang are the only prerequisites.

2 · Start the server

start.sh turns on the prefix cache, the web console, LAN sharing and a password in one command. Switching models later needs no restart: POST /v1/models/switch.

3 · Send a request

The example names glm-5.3-flash; GET /v1/models lists the id of whatever checkpoint is currently resident. Any OpenAI-compatible client works.