34 lines
665 B
Markdown
34 lines
665 B
Markdown
# TrueCluster
|
|
|
|
Prototype heterogeneous pipeline-parallel LLM inference cluster.
|
|
|
|
See [`SPEC.md`](SPEC.md).
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install -e .
|
|
```
|
|
|
|
## Run a cluster
|
|
|
|
```bash
|
|
truecluster cluster --model Qwen/Qwen2.5-0.5B-Instruct --max-nodes 1 --quant fp16
|
|
```
|
|
|
|
## Run a node
|
|
|
|
Nodes download/resolve the cluster model from HuggingFace themselves and load only the assigned layer range.
|
|
|
|
```bash
|
|
truecluster node --cluster-host 127.0.0.1 --cluster-port 7001 --device auto
|
|
```
|
|
|
|
## Generate
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8000/v1/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","prompt":"Hello","max_tokens":32}'
|
|
```
|