owen/longhaul.cpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implementing support for laguna arch

Browse Source
This commit is contained in:
Owen Qwen
2026-07-29 11:46:09 -05:00
parent 0bcb7ca59a
commit 68f5d104a8
8 changed files with 234 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/longhaul.md
+3 -4
View File
@@ -16,7 +16,7 @@ llama-cli \
`--longhaul-cache` is the expert cache budget in GiB. It is required when `--longhaul` is used. The same options are accepted by `llama-server`.
Longhaul may reduce `--ubatch-size` so that every expert selected by one graph segment can be present in the cache at the same time. The effective value is logged during context creation.
Longhaul may reduce `--ubatch-size` so that every expert selected by one graph segment can be present in the cache at the same time. The effective value is logged during context creation. If one token selects more experts than the cache has slots, the routed MoE computation is split into multiple stages and the partial results are summed. This permits smaller caches at the cost of additional graph work.
The normal startup warmup is skipped automatically in longhaul mode. Routed expert weights are not read until the first real decode.
@@ -25,14 +25,13 @@ The normal startup warmup is skipped automatically in longhaul mode. Routed expe
Longhaul currently requires:
- macOS with the Metal backend
- Qwen3.5 MoE architecture
- a single-file GGUF
- Qwen3.5 MoE or Laguna architecture
- all repeating model layers assigned to Metal
- text generation without embeddings or LoRA adapters
Longhaul does not restrict the GGUF quantization type. Individual tensor types must still be supported by Metal.
MTP/speculative decoding, split GGUF files, tensor validation during loading, vocabulary-only loading, and CPU or mixed CPU/Metal layer placement are not supported.
MTP/speculative decoding, tensor validation during loading, vocabulary-only loading, and CPU or mixed CPU/Metal layer placement are not supported. Both single-file and split GGUF models are supported; routed expert tensors are read from the shard that owns each tensor.
The cache budget covers the compact routed-expert tensors. It does not include dense weights, attention weights, the KV cache, graph allocations, or the temporary buffer used for one expert read. Disk reads bypass the macOS unified file cache where supported, avoiding a second long-lived copy of streamed weights in system RAM.