Add portable CPU longhaul support

docs/longhaul.md

@@ -11,11 +11,17 @@ llama-cli \
     --model /path/to/model.gguf \
     --longhaul \
     --longhaul-cache 2 \
-    --n-gpu-layers 99
+    --n-gpu-layers 0
 ```
 
 `--longhaul-cache` is the expert cache budget in GiB. It is required when `--longhaul` is used. The same options are accepted by `llama-server`.
 
+The command above runs every repeating layer on CPU and is supported on macOS,
+Linux, and Windows. On macOS, the existing all-Metal mode remains available by
+using `--n-gpu-layers 99` instead. Longhaul does not silently change device
+placement: an unsupported GPU or mixed CPU/GPU repeating-layer placement fails
+with guidance to use `--n-gpu-layers 0`.
+
 Longhaul may reduce `--ubatch-size` so that every expert selected by one graph segment can be present in the cache at the same time. The effective value is logged during context creation. If one token selects more experts than the cache has slots, the routed MoE computation is split into multiple stages and the partial results are summed. This permits smaller caches at the cost of additional graph work.
 
 The normal startup warmup is skipped automatically in longhaul mode. Routed expert weights are not read until the first real decode.
@@ -24,23 +30,38 @@ The normal startup warmup is skipped automatically in longhaul mode. Routed expe
 
 Longhaul currently requires:
 
-- macOS with the Metal backend
+- all repeating layers on CPU, or all repeating layers on Metal
 - Qwen3.5 MoE or Laguna architecture
-- all repeating model layers assigned to Metal
 - text generation without embeddings or LoRA adapters
 
-Longhaul does not restrict the GGUF quantization type. Individual tensor types must still be supported by Metal.
+CPU mode is available wherever the CPU backend is supported. Metal mode requires
+macOS. Longhaul does not restrict the GGUF quantization type; individual tensor
+types must still be supported by the selected compute backend.
 
-MTP/speculative decoding, tensor validation during loading, vocabulary-only loading, and CPU or mixed CPU/Metal layer placement are not supported. Both single-file and split GGUF models are supported; routed expert tensors are read from the shard that owns each tensor.
+MTP/speculative decoding, tensor validation during loading, vocabulary-only
+loading, and mixed CPU/GPU repeating-layer placement are not supported. Both
+single-file and split GGUF models are supported; routed expert tensors are read
+from the shard that owns each tensor.
 
-The cache budget covers the compact routed-expert tensors. It does not include dense weights, attention weights, the KV cache, graph allocations, or the temporary buffer used for one expert read. Disk reads bypass the macOS unified file cache where supported, avoiding a second long-lived copy of streamed weights in system RAM.
+The cache budget covers the compact routed-expert tensors. It does not include
+dense weights, attention weights, the KV cache, graph allocations, or temporary
+staging buffers. CPU expert caches use the standard directly writable tensor
+layout so individual slots can be replaced; optimized CPU buffer selection
+continues to apply to all non-streamed weights.
+
+File-cache control is best effort and is outside the explicit cache budget.
+macOS disables caching for streamed files where supported, and Linux advises the
+kernel to discard completed reads. Windows may retain file data in its system
+cache. Windows reads from one GGUF shard are serialized to preserve positional
+read correctness, while POSIX systems retain concurrent `pread` operations.
 
 This implementation synchronizes at each routed MoE layer to discover the selected experts, populate missing cache slots, and continue execution with cache-local expert IDs. Storage speed and expert reuse therefore have a large effect on generation speed.
 
 Expert IDs are planned as a batch at each synchronization point. Experts already
 needed by that batch are protected from eviction, duplicate IDs are loaded only
-once, and independent expert slices are read concurrently on shared-memory Metal
-devices. Private Metal buffers use a staged fallback.
+once, and independent expert slices are read concurrently where the platform
+supports positional reads. CPU and shared Metal buffers are populated directly;
+private Metal buffers use a staged fallback.
 
 ## Benchmarking prompt processing
 
@@ -51,7 +72,7 @@ llama-bench \
     --model /path/to/model.gguf \
     --load-mode longhaul \
     --longhaul-cache 2 \
-    --n-gpu-layers 99 \
+    --n-gpu-layers 0 \
     --n-prompt 2048 \
     --n-gen 0
 ```