Claude
dd2d52b5e3
When LLAMA_CACHED_MODEL is set but the model isn't present in the cache, find_cached.py was printing Python's None as the string "None", causing start.sh to pass "-m None" to llama-server. - find_cached.py: print an error to stderr and exit 1 when the model path cannot be resolved, instead of printing "None" - start.sh: capture find_cached.py output into a local variable, check the exit code and guard against empty output before constructing CACHED_LLAMA_ARGS; also quote the env-var expansions to handle spaces https://claude.ai/code/session_011ny5CFYnrzPbRneSzio5CR
Serverless llama.cpp inference worker for RunPod
This repository contains a serverless inference worker for running llama.cpp models on RunPod. It uses the llama-server image to provide an API for interacting with the models.
The following OpenAI API endpoints are supported:
v1/modelsv1/chat/completionsv1/completions
Streaming responses is also supported.
Important! This project is still relatively new. Please open a new issue if you encounter any problems in order to get help.
This is a fork of SvenBrnn's runpod-worker-ollama.
Setup
To get the best performance out of this worker, it is recommended to use cached models. Please see the cached models documentation for more information, this is highly recommended and will save many resources.
Configuration
The worker can be configured via environment variables set in the RunPod hub configuration:
LLAMA_SERVER_CMD_ARGS: Command line arguments (argv) for thellama-serverbinary. Example:-hf /path/to/model.gguf:Q4_K_M --ctx-size 4096. IMPORTANT: Please do not define the port argument here, as the worker will always use port3098automatically.MAX_CONCURRENCY: Maximum number of concurrent requests the worker can handle. Default is8.
License
Please see the LICENSE file for more information.