TrueCluster

Prototype heterogeneous pipeline-parallel LLM inference cluster.

See SPEC.md.

Install

pip install -e .

Run a cluster

truecluster cluster --model Qwen/Qwen2.5-0.5B-Instruct --max-nodes 1 --quant fp16

Run a node

Nodes download/resolve the cluster model from HuggingFace themselves and load only the assigned layer range.

truecluster node --cluster-host 127.0.0.1 --cluster-port 7001 --device auto

Generate

curl http://127.0.0.1:8000/v1/completions \
  -H 'content-type: application/json' \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","prompt":"Hello","max_tokens":32}'