Add RPC longhaul paging
This commit is contained in:
+11
-1
@@ -36,6 +36,17 @@ flowchart TD
|
||||
By default, `ggml-rpc-server` exposes all available accelerator devices on the host.
|
||||
If there are no accelerators, it exposes a single `CPU` device.
|
||||
|
||||
For server-resident longhaul MoE paging, place the same GGUF shards on the RPC
|
||||
host and point the server at their directory:
|
||||
|
||||
```sh
|
||||
$ bin/ggml-rpc-server --device MTL0 --longhaul-root /path/to/model-directory
|
||||
```
|
||||
|
||||
The client can then use `--rpc`, `--longhaul`, and `--longhaul-cache` together.
|
||||
RPC longhaul v1 requires one remote Metal device. Expert cache misses are read
|
||||
from the RPC host's local files rather than uploaded by the client.
|
||||
|
||||
## Usage
|
||||
|
||||
### Remote hosts
|
||||
@@ -107,4 +118,3 @@ Use the `GGML_RPC_DEBUG` environment variable to enable debug messages from `ggm
|
||||
```bash
|
||||
$ GGML_RPC_DEBUG=1 bin/ggml-rpc-server
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user