Skip to main content

Router control plane

The router control plane is the local runtime layer for llama.cpp models. It maps short model ids to GGUF files, starts llama-server in router mode, and exposes load/unload/status commands through ,llama-cpp.

Mental model​

models.ini is the preset: it names the models and their per-model defaults. ,llama-cpp is the operator interface: it manages the shared server lifecycle and calls the model API.

The shipped preset defines these model ids. They inherit shared [*] defaults. The router loads one at a time on demand.

Using it​

Router preset​

llama.cpp model routing and per-model defaults live in an INI preset:

The shipped preset defines these short model ids:

IDGGUF pathUse
nemotron-3.5~/.llama.cpp/models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-UD-Q4_K_XL.ggufUnsloth Nemotron agentic model
qwen3.6-35b-a3b~/.llama.cpp/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.ggufUnsloth Qwen3.6 35B A3B + dest-renamed mmproj
qwen3.6-35b-a3b-instructsame Qwen3.6-35B-A3B-UD-Q5_K_XL.ggufNon-thinking profile of the same weights

They inherit shared [*] defaults:

  • ctx-size=262144
  • Metal offload
  • flash attention
  • Jinja chat templates
  • q8 KV cache
  • reasoning=auto
  • nemotron-3.5 forces reasoning=off (thinking off by default for speed; a request's chat_template_kwargs.enable_thinking still turns it on), sets in-model NextN MTP (spec-type=draft-mtp, spec-draft-n-max=2) plus Unsloth thinking sampling (temp=0.6, top-p=0.95, min-p=0.01)
  • qwen3.6-35b-a3b keeps reasoning=auto with thinking sampling (temp=1.0, top-p=0.95, top-k=20, min-p=0, presence-penalty=1.5, repeat-penalty=1.0)
  • qwen3.6-35b-a3b-instruct forces reasoning=off with instruct sampling (temp=0.7, top-p=0.80, top-k=20, min-p=0, presence-penalty=1.5, repeat-penalty=1.0); same weights, loaded on demand as a separate preset

Switch with ,llama-cpp load <id> / ,llama-cpp unload <id>.

The served default ctx-size is 262144 for Nemotron and Qwen3.6; Claude Code selects a Qwen3.6-specific settings file with autoCompactWindow=200000.

,llama-cpp serve
curl -s http://localhost:8080/models | python3 -m json.tool

Optional llama-server binary selection​

serve and run retain the optional PrismML fork build path for compatible future model weights. The current roster has no fork-only preset.

Both serve and run resolve the server binary in the same order: $LLAMA_CPP_SERVER_BIN when set, else $LLAMA_CPP_PRISM_ROOT/bin/llama-server when it is an executable file, else llama-server from PATH. Homebrew llama.cpp remains installed and untouched as that PATH fallback, so every other model id keeps serving normally whether or not the fork is built. See Launchers for the build command and layout.

Shared lifecycle​

The four *-llama-cpp launchers use ,llama-cpp run -- <command>. Each process holds a lease for the configured host and port:

  1. A reachable router is joined. A router started manually with ,llama-cpp serve is never stopped by the lease manager.
  2. An absent loopback router is started from the configured preset and recorded with its PID and process-start identity.
  3. The last consumer schedules that recorded process to stop after LLAMA_CPP_GRACE_SECONDS (default 600). A new consumer during grace cancels the pending shutdown and reuses the loaded router; 0 restores immediate shutdown.

Automatic startup is limited to loopback hosts. A missing non-loopback router fails closed because the launcher cannot safely start or own a remote process. run forwards SIGINT, SIGTERM, and SIGHUP (a closed terminal, for example tmux kill-session) to the command, waits for it, and still releases its lease; repeated signals are forwarded without interrupting that wait, and an inherited SIGHUP ignore (nohup) is left to the command. Stale lease files left by an uncatchable process exit are pruned on the next lifecycle operation.

Use ,llama-cpp stop to end a lifecycle-owned router during its grace period. It never stops a manually started router. If a harness still holds a lease, the command refuses to interrupt it; ,llama-cpp stop --force is the explicit override and will break those active sessions.

Model-level control plane (,llama-cpp)​

This repo ships a thin wrapper around llama-server router mode and its model API:

,llama-cpp serve                      # start llama-server router mode
,llama-cpp run -- <command> [args...] # acquire a shared router lease
,llama-cpp stop [--force]             # stop the lifecycle-owned router now
,llama-cpp status                     # loaded/unloaded state
,llama-cpp load <model-id> [<id> ...] # POST /models/load
,llama-cpp unload <model-id> [<id> ...]
,llama-cpp unload --all
,llama-cpp build-prism                # build/install the PrismML fork server

Reference​

VariableDefaultPurpose
LLAMA_CPP_HOST127.0.0.1llama.cpp host
LLAMA_CPP_PORT8080llama.cpp port
LLAMA_CPP_API_KEYno auth header unless setoptional server and request key
LLAMA_CPP_MODELS_PRESET~/.config/llama.cpp/models.inialternate model preset
LLAMA_CPP_LIFECYCLE_DIR~/.local/state/llama-cpp/lifecyclelease, owner, router-log, and shutdown state
LLAMA_CPP_GRACE_SECONDS600delay after the last lease; 0 stops at once
LLAMA_CPP_PRISM_ROOT~/.llama.cpp/prismPrismML fork checkout and installed binaries
LLAMA_CPP_SERVER_BINunsetexplicit llama-server path or PATH name

,llama-cpp respects LLAMA_CPP_HOST / LLAMA_CPP_PORT / LLAMA_CPP_API_KEY / LLAMA_CPP_MODELS_PRESET (defaults: 127.0.0.1:8080, no auth header unless LLAMA_CPP_API_KEY is set, preset at ~/.config/llama.cpp/models.ini).

Internals​

The ,llama-cpp command is a thin launcher. Its command library implements serve/run/stop/status/load/unload/build-prism; lifecycle.py owns locking, leases, process identity, and cleanup. Its fish completion provides context-aware subcommand + model-id completions.

Sources and verification​