98 lines
2.4 KiB
Markdown
98 lines
2.4 KiB
Markdown
# TrueCluster
|
|
|
|
Prototype heterogeneous pipeline-parallel LLM inference cluster.
|
|
|
|
See [`SPEC.md`](SPEC.md).
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install -e .
|
|
```
|
|
|
|
### Nvidia CUDA install
|
|
|
|
If a node has an Nvidia GPU, it must install a CUDA-enabled PyTorch build. If you see:
|
|
|
|
```text
|
|
Torch not compiled with CUDA enabled
|
|
```
|
|
|
|
then the node installed the CPU-only PyTorch package.
|
|
|
|
Recommended fix on a macOS/Linux Nvidia node:
|
|
|
|
```bash
|
|
source .venv/bin/activate
|
|
pip uninstall -y torch torchvision torchaudio
|
|
pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision torchaudio
|
|
pip install -e .
|
|
```
|
|
|
|
Recommended fix on a Windows Nvidia node using PowerShell:
|
|
|
|
```powershell
|
|
.\.venv\Scripts\Activate.ps1
|
|
python -m pip uninstall -y torch torchvision torchaudio
|
|
python -m pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision torchaudio
|
|
python -m pip install -e .
|
|
```
|
|
|
|
If PowerShell blocks venv activation, run this once in the same PowerShell window:
|
|
|
|
```powershell
|
|
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
|
|
.\.venv\Scripts\Activate.ps1
|
|
```
|
|
|
|
Windows Command Prompt alternative:
|
|
|
|
```bat
|
|
.venv\Scripts\activate.bat
|
|
python -m pip uninstall -y torch torchvision torchaudio
|
|
python -m pip install --index-url https://download.pytorch.org/whl/cu121 torch torchvision torchaudio
|
|
python -m pip install -e .
|
|
```
|
|
|
|
For newer CUDA builds, PyTorch may also provide `cu124` or `cu126` wheels. Check https://pytorch.org/get-started/locally/ if `cu121` is not appropriate.
|
|
|
|
Verify CUDA support:
|
|
|
|
```bash
|
|
python - <<'PY'
|
|
import torch
|
|
print('torch:', torch.__version__)
|
|
print('cuda available:', torch.cuda.is_available())
|
|
print('cuda version:', torch.version.cuda)
|
|
print('gpu:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else None)
|
|
PY
|
|
```
|
|
|
|
Then run the node with:
|
|
|
|
```bash
|
|
truecluster node --cluster-host YOUR_CLUSTER_IP --cluster-port 7001 --device cuda:0
|
|
```
|
|
|
|
## Run a cluster
|
|
|
|
```bash
|
|
truecluster cluster --model Qwen/Qwen2.5-0.5B-Instruct --max-nodes 1 --quant fp16
|
|
```
|
|
|
|
## Run a node
|
|
|
|
Nodes download/resolve the cluster model from HuggingFace themselves and load only the assigned layer range.
|
|
|
|
```bash
|
|
truecluster node --cluster-host 127.0.0.1 --cluster-port 7001 --device auto
|
|
```
|
|
|
|
## Generate
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8000/v1/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","prompt":"Hello","max_tokens":32}'
|
|
```
|