owen/worker-vllm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
442 Commits 1 Branch 69 Tags
aef1187a30f0e5e10f72f94836c322223bab14b7
Commit Graph
56 Commits
Author SHA1 Message Date
Tim Pietrusky 121a3dd44b fix: parse value for RAW_OPENAI_OUTPUT correctly 2025-08-13 12:20:44 +02:00
Jhenner Tigreros fb0c030797 fix initialization on openaiservingmodels 2025-08-07 16:40:06 -05:00
Tim Pietrusky 1e9a731380 Fix Mistral tokenizer initialization: let vLLM handle tokenizer for mistral models 2025-06-14 15:31:56 +02:00
pandyamarut 70cd1c8113 update vllm version 0.9.0 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2025-06-05 11:59:15 -07:00
RedHitMark 084d000324 fix lora and multi-lora 2025-03-14 21:45:09 +01:00
Marut PandyaandGitHub 2b1d618287 Revert "Enabling model caching." 2025-02-18 10:45:47 -08:00
Marut PandyaandGitHub 6fc770415d Merge pull request #157 from runpod-workers/m-c 
Enabling model caching.
 2025-02-06 11:52:19 -08:00
pandyamarut 30dd7c1eb5 update engine.py 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2025-01-28 21:40:00 -08:00
pandyamarut dc8c88027a update serving classes 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2025-01-28 20:58:07 -08:00
pandyamarut c703254f71 dynamic lora loading 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2025-01-28 19:00:57 -08:00
Marut PandyaandGitHub 3f0a20d28e Merge pull request #141 from runpod-workers/main 
Rebase
 2025-01-02 20:49:48 -08:00
pandyamarut 66ea8b1110 update engine 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-12-30 17:13:03 -08:00
kldzj b7787d8ca8 fix: remove unused import 2024-12-06 16:29:06 +01:00
kldzj b1fca5d257 fix: move tool flags away from engine args 2024-12-06 16:07:23 +01:00
kldzj 3e86d16892 fix: openai tool calling 2024-12-06 13:08:48 +01:00
pandyamarut 6c6bf50379 update env 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-11-22 15:14:57 -08:00
pandyamarut 27a2ee5754 add model cache 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-11-22 13:11:12 -08:00
Sven Knoblauch 677a01e8f3 update code for case of no lora adapter 2024-10-09 14:51:33 +02:00
Sven Knoblauch 5cd12ba331 add changes for lora adapter support and /v1/models endpoint 2024-10-09 11:01:12 +02:00
pandyamarut 1420091588 update vll 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-09-27 19:23:50 -07:00
pandyamarut 814f50af38 fix oai completion api error 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-09-06 11:03:47 -07:00
pandyamarut 9cb9336cf5 update vllm version 0.5.4 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-08-09 12:01:42 -07:00
pandyamarut f3534a4ea7 fix openai compat 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-07-27 15:49:45 -07:00
pandyamarut bd96b5e0de update v0.5.3.post1 
Signed-off-by: pandyamarut <pandyamarut@gmail.com>
 2024-07-25 21:29:45 -07:00
alpayariyak 5bd6f3a75e 0.5.3, any vllm arg as env var, refactor and fixes, moving away from building separate image from vLLM fork 2024-07-25 12:41:48 -07:00
alpayariyak a08d83f600 Allow any vLLM engine args as env vars, refactor 2024-07-02 19:44:01 +00:00
alpayariyak bad5ddd892 Fix vLLM 0.4.1 bug for OpenAI /models route 2024-06-10 20:38:49 +00:00
Alpay Ariyakandalpayariyak 874379a0c5 1.0.0preview update for Llama 3 support and more (vLLM 0.3.3 -> 0.4.2) (#62) 2024-05-09 00:42:08 -04:00
alpayariyak d25b6f9628 Fix sampling params 2024-03-12 22:15:57 +00:00
alpayariyak c8ee100d80 Small refactor 2024-03-06 17:08:57 +00:00
alpayariyak 985bbf1cb5 Fix Multi-GPU, tokenizer trust remote code 2024-02-24 03:18:25 +00:00
alpayariyak 708f68d7f8 Final Bug fixes, configurable oai response role, served model name override, documentation 2024-02-23 04:14:03 +00:00
alpayariyak b42d45ce0f Bug fixes, refactors 2024-02-23 03:46:47 +00:00
alpayariyak aed0408f19 Documentation for 0.3.0, small fixes and changes 2024-02-21 19:28:59 -05:00
alpayariyak e191149259 OpenAI Compatibility, Dynamic Batching, Refactor 2024-02-21 04:36:04 +00:00
alpayariyak a94ef66f71 Dynamic Batch Size [needs refactor] 2024-02-06 03:47:27 +00:00
alpayariyak fef8c81cb9 OpenAI Compatible worker, Refactor 2024-02-06 02:44:14 +00:00
alpayariyak 15b06bb687 Merge branch 'main' into openai-sse-output 2024-02-02 20:03:29 -05:00
alpayariyak 8de10468dd Working tokenizer and model download fix 
Fix handler startup
 2024-02-02 19:55:35 -05:00
alpayariyak b1720a154d Added download of model extras into weights folder, separate download of tokenizer, making engine.py utilize downloaded tokenizer, model and tokenizer revision 2024-01-31 22:57:32 -05:00
alpayariyak afa33a2875 Handle errors 2024-02-01 03:01:51 +00:00
alpayariyak 3bbcf0021b Fix: Yield if tokens left in batch 
Temp: default model for testing image
 2024-02-01 00:52:39 +00:00
alpayariyak dab8bad906 OpenAI Chat Completions Stream 2024-01-31 23:27:31 +00:00
alpayariyak 3e2cd080a2 Fix Tensor Parallel 2024-01-31 05:12:58 +00:00
alpayariyak 46eee12819 Simplify Tensor Parallel 2024-01-31 03:28:02 +00:00
alpayariyak 12d6f0778e Fixed Model bake-in, added Custom Chat Templates, Custom Tokenizer 2024-01-31 03:13:21 +00:00
alpayariyak 9fc8e1e54c Non-streaming OpenAI Chat Completions 2024-01-25 23:24:18 -05:00
alpayariyak 4cebe66b36 0.2.0 Release 
- You no longer need a linux-based machine or NVIDIA GPUs to build the worker.
- Over 3x lighter Docker image size.
- OpenAI Chat Completion output format (optional to use).
- Extremely fast image build time.
- Docker Secrets-protected Hugging Face token support for building the image with a model baked in without exposing your token.
- Support for `n` and `best_of` sampling parameters, which allow you to generate multiple responses from a single prompt.
- New environment variables for various configuration.
- vLLM Version: 0.2.7
 2024-01-25 20:49:15 -05:00
alpayariyak 584852f0f6 Docker-Protected HF Token, Refactor, Better Documentation 2024-01-18 18:20:51 -05:00
Alpay AriyakandGitHub ed48093792 bug fix 2024-01-16 18:38:26 -05:00