longhaul.cpp/docs/longhaul.md

Longhaul MoE loading

Longhaul mode keeps the non-expert model weights resident and loads routed MoE expert slices from the GGUF file as they are needed. The expert cache has a fixed budget and uses least-recently-used replacement independently for each layer.

This mode is intended for running a model whose full expert weights do not fit in unified memory. It trades throughput and latency for a smaller resident model allocation.

Usage

llama-cli \
    --model /path/to/model.gguf \
    --longhaul \
    --longhaul-cache 2 \
    --n-gpu-layers 99

--longhaul-cache is the expert cache budget in GiB. It is required when --longhaul is used. The same options are accepted by llama-server.

RPC mode

RPC longhaul keeps the paging loop on the accelerator host so expert payloads do not cross the network during generation. Build both hosts with -DGGML_RPC=ON. Place the same GGUF shard files on both hosts, preserving their basenames, then start the Metal host with the directory containing those shards:

ggml-rpc-server \
    --device MTL0 \
    --longhaul-root /path/to/model-directory \
    --host 192.168.1.20

Start the client normally:

llama-server \
    --model /local/path/model.gguf \
    --rpc 192.168.1.20:50052 \
    --longhaul \
    --longhaul-cache 2 \
    --n-gpu-layers 99

RPC longhaul currently requires all repeating layers on one remote Metal device. The server validates each shard basename, size, GGUF metadata layout digest, and every registered expert range before decoding. An older server, a server without --longhaul-root, or a shard mismatch is a fatal model-load error; the client does not fall back to sending expert slices over RPC.

--longhaul-root grants connected RPC clients read access to registered byte ranges in files in that directory. The RPC server remains experimental and insecure and must not be exposed to an untrusted network.

Longhaul may reduce --ubatch-size so that every expert selected by one graph segment can be present in the cache at the same time. The effective value is logged during context creation. If one token selects more experts than the cache has slots, the routed MoE computation is split into multiple stages and the partial results are summed. This permits smaller caches at the cost of additional graph work.

The normal startup warmup is skipped automatically in longhaul mode. Routed expert weights are not read until the first real decode.

Current scope

Longhaul currently requires:

• the Metal backend, either local on macOS or on one RPC host
• Qwen3.5 MoE or Laguna architecture
• all repeating model layers assigned to local Metal or one RPC Metal device
• text generation without embeddings or LoRA adapters

Longhaul does not restrict the GGUF quantization type. Individual tensor types must still be supported by Metal.

MTP/speculative decoding, tensor validation during loading, vocabulary-only loading, and CPU or mixed CPU/Metal layer placement are not supported. Both single-file and split GGUF models are supported; routed expert tensors are read from the shard that owns each tensor.

The cache budget covers the compact routed-expert tensors. It does not include dense weights, attention weights, the KV cache, graph allocations, or the temporary buffer used for one expert read. Disk reads bypass the macOS unified file cache where supported, avoiding a second long-lived copy of streamed weights in system RAM.

This implementation synchronizes at each routed MoE layer to discover the selected experts, populate missing cache slots, and continue execution with cache-local expert IDs. Storage speed and expert reuse therefore have a large effect on generation speed.

Expert IDs are planned as a batch at each synchronization point. Experts already needed by that batch are protected from eviction, duplicate IDs are loaded only once, and independent expert slices are read concurrently on shared-memory Metal devices. Private Metal buffers use a staged fallback.

Benchmarking prompt processing

llama-bench accepts the longhaul load mode and cache budget:

llama-bench \
    --model /path/to/model.gguf \
    --load-mode longhaul \
    --longhaul-cache 2 \
    --n-gpu-layers 99 \
    --n-prompt 2048 \
    --n-gen 0

Use --no-warmup --repetitions 1 in separate processes to measure a cold expert cache. Leave warmup enabled to measure steady-state cache reuse.