owen/inference-worker
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix several major bugs in startup script

Browse Source
This commit is contained in:
mags0ft
2025-11-19 17:47:06 +01:00
parent 36a8b32d1e
commit 40d4097799
5 changed files with 28 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3 -3
View File
@@ -4,8 +4,6 @@
# Serverless llama.cpp inference worker for RunPod
[![Runpod badge](https://api.runpod.io/badge/Jacob-ML/inference-worker)](https://console.runpod.io/hub/Jacob-ML/inference-worker)
This repository contains a serverless inference worker for running llama.cpp models on RunPod. It uses the `llama-server` image to provide an API for interacting with the models.
The following OpenAI API endpoints are supported:
@@ -24,9 +22,11 @@ Make sure your RunPod worker has access to the network volume, i.e. is located i
The worker can be configured via environment variables set in the RunPod hub configuration:
- `LLAMA_SERVER_CMD_ARGS`: Command line arguments (argv) for the `llama-server` binary. Example: `-hf /path/to/model.gguf:Q4_K_M -ctx_size 4096`. **IMPORTANT**: Please do not define the port argument here, as the worker will always use port `3098` automatically.
- `LLAMA_SERVER_CMD_ARGS`: Command line arguments (argv) for the `llama-server` binary. Example: `-hf /path/to/model.gguf:Q4_K_M --ctx-size 4096`. **IMPORTANT**: Please do not define the port argument here, as the worker will always use port `3098` automatically.
- `MAX_CONCURRENCY`: Maximum number of concurrent requests the worker can handle. Default is `8`.
## License
Please see the [LICENSE](./LICENSE) file for more information.
[![Runpod badge](https://api.runpod.io/badge/Jacob-ML/inference-worker)](https://console.runpod.io/hub/Jacob-ML/inference-worker)