# Longhaul MoE loading

Longhaul mode keeps the non-expert model weights resident and loads routed MoE expert slices from the GGUF file as they are needed. The expert cache has a fixed budget and uses least-recently-used replacement independently for each layer.

This mode is intended for running a model whose full expert weights do not fit in unified memory. It trades throughput and latency for a smaller resident model allocation.

## Usage

```sh
llama-cli \
    --model /path/to/model.gguf \
    --longhaul \
    --longhaul-cache 0.5 \
    --n-gpu-layers 99
```

`--longhaul-cache` is the expert cache budget in GiB and accepts decimal values.
It is required when `--longhaul` is used. The same options are accepted by
`llama-server`.

Longhaul may reduce `--ubatch-size` so that every expert selected by one graph segment can be present in the cache at the same time. The effective value is logged during context creation. If one token selects more experts than the cache has slots, the routed MoE computation is split into multiple stages and the partial results are summed. This permits smaller caches at the cost of additional graph work.

The normal startup warmup is skipped automatically in longhaul mode. Routed expert weights are not read until the first real decode.

## Current scope

Longhaul currently requires:

- macOS with Metal, or Linux with Vulkan
- Qwen3.x MoE or Laguna architecture
- all repeating model layers assigned to one supported GPU
- text generation without embeddings or LoRA adapters

Longhaul does not restrict the GGUF quantization type. Individual tensor types
must still be supported by the selected GPU backend.

MTP/speculative decoding, tensor validation during loading, vocabulary-only
loading, multi-GPU placement, and CPU or mixed CPU/GPU layer placement are not
supported. Both single-file and split GGUF models are supported; routed expert
tensors are read from the shard that owns each tensor.

The cache budget covers the compact routed-expert tensors. It does not include
dense weights, attention weights, the KV cache, graph allocations, or temporary
staging buffers. Disk reads bypass the macOS unified file cache where supported.
On Linux, completed streamed reads are released from the kernel page cache. Both
avoid a second long-lived copy of streamed weights in system RAM.

This implementation synchronizes at each routed MoE layer to discover the selected experts, populate missing cache slots, and continue execution with cache-local expert IDs. Storage speed and expert reuse therefore have a large effect on generation speed.

Expert IDs are planned as a batch at each synchronization point. Experts already
needed by that batch are protected from eviction, duplicate IDs are loaded only
once, and independent expert slices are read concurrently. Shared host or Metal
buffers can be populated directly. Private Metal and Vulkan device buffers use
staged uploads that are synchronized before routed computation continues.

## Arch Linux and a 4 GiB GTX 1050 Ti

Install the Vulkan build dependencies and confirm that the NVIDIA GPU is visible:

```sh
sudo pacman -S --needed base-devel cmake ninja shaderc spirv-headers \
    vulkan-headers vulkan-icd-loader vulkan-tools
vulkaninfo --summary
```

A GTX 10-series card needs a Vulkan driver that still supports Pascal. Driver
installation is system-specific; do not continue until `vulkaninfo` lists the
GTX 1050 Ti.

Build Longhaul with Vulkan and list the runtime device names:

```sh
cmake -S . -B build-vulkan -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_VULKAN=ON
cmake --build build-vulkan --target llama-cli llama-bench test-longhaul
./build-vulkan/bin/llama-cli --list-devices
```

Use the `VulkanN` name corresponding to the discrete NVIDIA card. This matters
on systems that also expose an integrated Radeon GPU.

For `Qwen3.6-35B-A3B-UD-IQ4_XS.gguf`, the non-expert weights use about 2.38 GiB
and each cache slot uses about 56.5 MiB. A 0.5 GiB cache provides nine slots for
the model's eight selected experts:

```sh
./build-vulkan/bin/llama-cli \
    --model /path/to/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
    --device VulkanN \
    --n-gpu-layers 99 \
    --longhaul \
    --longhaul-cache 0.5 \
    --ctx-size 512 \
    --no-kv-offload \
    --prompt "Write a short hello message." \
    --n-predict 32
```

Keep the model on the fastest available drive. A hard drive works correctly,
but random expert reads can make generation substantially slower. If Vulkan
runs out of memory while the desktop is active, close GPU-heavy applications
before reducing the cache below eight slots.

## Benchmarking prompt processing

`llama-bench` accepts the longhaul load mode and cache budget:

```sh
llama-bench \
    --model /path/to/model.gguf \
    --load-mode longhaul \
    --longhaul-cache 0.5 \
    --n-gpu-layers 99 \
    --n-prompt 2048 \
    --n-gen 0
```

Use `--no-warmup --repetitions 1` in separate processes to measure a cold expert
cache. Leave warmup enabled to measure steady-state cache reuse.