Add RPC longhaul paging

This commit is contained in:
Owen Qwen
2026-07-30 14:39:47 -05:00
parent a673afad85
commit d080ba3d65
13 changed files with 1301 additions and 43 deletions
+11 -1
View File
@@ -36,6 +36,17 @@ flowchart TD
By default, `ggml-rpc-server` exposes all available accelerator devices on the host.
If there are no accelerators, it exposes a single `CPU` device.
For server-resident longhaul MoE paging, place the same GGUF shards on the RPC
host and point the server at their directory:
```sh
$ bin/ggml-rpc-server --device MTL0 --longhaul-root /path/to/model-directory
```
The client can then use `--rpc`, `--longhaul`, and `--longhaul-cache` together.
RPC longhaul v1 requires one remote Metal device. Expert cache misses are read
from the RPC host's local files rather than uploaded by the client.
## Usage
### Remote hosts
@@ -107,4 +118,3 @@ Use the `GGML_RPC_DEBUG` environment variable to enable debug messages from `ggm
```bash
$ GGML_RPC_DEBUG=1 bin/ggml-rpc-server
```