owen/inference-worker
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

add caching support for models in start.sh and update documentation

Browse Source
This commit is contained in:
mags0ft
2025-12-26 14:49:03 +01:00
parent 0a59d57025
commit 0c814177fe
4 changed files with 104 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -2
View File
@@ -19,8 +19,7 @@ Streaming responses is also supported.
## Setup
For the setup to work best, it is recommended to use a network volume attached to all workers which stores the model GGUFs and then reference those files in the launch arguments.
Make sure your RunPod worker has access to the network volume, i.e. is located in the correct data center.
To get the best performance out of this worker, it is recommended to use cached models. Please see the [cached models documentation](./docs/cached.md) for more information, this is **highly recommended and will save many resources**.
## Configuration