2.1 KiB
Longhaul MoE loading
Longhaul mode keeps the non-expert model weights resident and loads routed MoE expert slices from the GGUF file as they are needed. The expert cache has a fixed budget and uses least-recently-used replacement independently for each layer.
This mode is intended for running a model whose full expert weights do not fit in unified memory. It trades throughput and latency for a smaller resident model allocation.
Usage
llama-cli \
--model /path/to/model.gguf \
--longhaul \
--longhaul-cache 2 \
--n-gpu-layers 99
--longhaul-cache is the expert cache budget in GiB. It is required when --longhaul is used. The same options are accepted by llama-server.
Longhaul may reduce --ubatch-size so that every expert selected by one graph segment can be present in the cache at the same time. The effective value is logged during context creation.
The normal startup warmup is skipped automatically in longhaul mode. Routed expert weights are not read until the first real decode.
Current scope
Longhaul currently requires:
- macOS with the Metal backend
- Qwen3.5 MoE architecture
- a single-file GGUF
- all repeating model layers assigned to Metal
- text generation without embeddings or LoRA adapters
Longhaul does not restrict the GGUF quantization type. Individual tensor types must still be supported by Metal.
MTP/speculative decoding, split GGUF files, tensor validation during loading, vocabulary-only loading, and CPU or mixed CPU/Metal layer placement are not supported.
The cache budget covers the compact routed-expert tensors. It does not include dense weights, attention weights, the KV cache, graph allocations, or the temporary buffer used for one expert read. Disk reads bypass the macOS unified file cache where supported, avoiding a second long-lived copy of streamed weights in system RAM.
This implementation synchronizes at each routed MoE layer to discover the selected experts, populate missing cache slots, and continue execution with cache-local expert IDs. Storage speed and expert reuse therefore have a large effect on generation speed.