diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..295faa0 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ + +runpod.toml diff --git a/README.md b/README.md index 59842c3..9813ad2 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,7 @@

vLLM Endpoint | Serverless Worker

-[![CI | Test Worker](https://github.com/runpod-workers/worker-template/actions/workflows/CI-test_worker.yml/badge.svg)](https://github.com/runpod-workers/worker-template/actions/workflows/CI-test_worker.yml) -  -[![Docker Image](https://github.com/runpod-workers/worker-template/actions/workflows/CD-docker_dev.yml/badge.svg)](https://github.com/runpod-workers/worker-template/actions/workflows/CD-docker_dev.yml) +[![CD | Docker-Build-Release](https://github.com/runpod-workers/worker-vllm/actions/workflows/docker-build-release.yml/badge.svg)](https://github.com/runpod-workers/worker-vllm/actions/workflows/docker-build-release.yml) 🚀 | This serverless worker utilizes vLLM behind the scenes and is integrated into RunPod's serverless environment. It supports dynamic auto-scaling using the built-in RunPod autoscaling feature. @@ -12,7 +10,7 @@ ## Setting up the Serverless Worker ### Option 1:Deploy Any Model Using Pre-Built Docker Image -We now offer a pre-built Docker Image for the vLLM Worker that you can configure entirely with Environment Variables when creating the RunPod Serverless Endpoint: +We now offer a pre-built Docker Image for the vLLM Worker that you can configure entirely with Environment Variables when creating the RunPod Serverless Endpoint:
@@ -23,7 +21,7 @@ We now offer a pre-built Docker Image for the vLLM Worker that you can configure #### Environment Variables - **Required**: - `MODEL_NAME`: Hugging Face Model Repository (e.g., `openchat/openchat_3.5`). - + - **Optional**: - `MODEL_BASE_PATH`: Model storage directory (default: `/runpod-volume`). - `HF_TOKEN`: Hugging Face token for private and gated models (e.g., Llama, Falcon). @@ -48,25 +46,25 @@ To build an image with the model baked in, you must specify the following docker `sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .` ### Compatible Models -- LLaMA & LLaMA-2 +- LLaMA & LLaMA-2 - Mistral - Mixtral (Mistral MoE) - Yi - ChatGLM - Phi -- MPT -- OPT -- Qwen -- Aquila & Aquila2 +- MPT +- OPT +- Qwen +- Aquila & Aquila2 - Baichuan -- BLOOM -- Falcon +- BLOOM +- Falcon - GPT-2 - GPT BigCode - GPT-J - GPT-NeoX - InternLM - + And any other models supported by vLLM 0.2.4. @@ -74,37 +72,37 @@ Ensure that you have Docker installed and properly set up before running the doc ## Model Inputs -| Argument | Type | Default | Description | -|--------------------|-----------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| prompt | str | | Prompt string to generate text based on. | -| sampling_params | dict | {} | Sampling parameters to control the generation, like temperature, top_p, etc. | -| streaming | bool | False | Whether to enable streaming of output. If True, responses are streamed as they are generated. | -| batch_size | int | DEFAULT_BATCH_SIZE | The number of responses to generate in one batch. Only applicable +| Argument | Type | Default | Description | +|-----------------|------|--------------------|-----------------------------------------------------------------------------------------------| +| prompt | str | | Prompt string to generate text based on. | +| sampling_params | dict | {} | Sampling parameters to control the generation, like temperature, top_p, etc. | +| streaming | bool | False | Whether to enable streaming of output. If True, responses are streamed as they are generated. | +| batch_size | int | DEFAULT_BATCH_SIZE | The number of responses to generate in one batch. Only applicable | ### Sampling Parameters -| Argument | Type | Default | Description | -|---------------------------------|--------------------------------|-----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| n | int | 1 | Number of output sequences to return for the given prompt. | -| best_of | Optional[int] | None | Number of output sequences generated from the prompt. The top `n` sequences are returned from these `best_of` sequences. Must be ≥ `n`. Treated as beam width in beam search. Default is `n`. | -| presence_penalty | float | 0.0 | Penalizes new tokens based on their presence in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition. | -| frequency_penalty | float | 0.0 | Penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition. | -| repetition_penalty | float | 1.0 | Penalizes new tokens based on their appearance in the prompt and generated text. Values > 1 encourage new tokens, values < 1 encourage repetition. | -| temperature | float | 1.0 | Controls the randomness of sampling. Lower values make it more deterministic, higher values make it more random. Zero means greedy sampling. | -| top_p | float | 1.0 | Controls the cumulative probability of top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. | -| top_k | int | -1 | Controls the number of top tokens to consider. Set to -1 to consider all tokens. | -| min_p | float | 0.0 | Represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable. | -| use_beam_search | bool | False | Whether to use beam search instead of sampling. | -| length_penalty | float | 1.0 | Penalizes sequences based on their length. Used in beam search. | -| early_stopping | Union[bool, str] | False | Controls stopping condition in beam search. Can be `True`, `False`, or `"never"`. | -| stop | Union[None, str, List[str]] | None | List of strings that stop generation when produced. Output will not contain these strings. | -| stop_token_ids | Optional[List[int]] | None | List of token IDs that stop generation when produced. Output contains these tokens unless they are special tokens. | -| ignore_eos | bool | False | Whether to ignore the End-Of-Sequence token and continue generating tokens after its generation. | -| max_tokens | int | 16 | Maximum number of tokens to generate per output sequence. | -| logprobs | Optional[int] | None | Number of log probabilities to return per output token. | -| prompt_logprobs | Optional[int] | None | Number of log probabilities to return per prompt token. | -| skip_special_tokens | bool | True | Whether to skip special tokens in the output. | -| spaces_between_special_tokens | bool | True | Whether to add spaces between special tokens in the output. | - +| Argument | Type | Default | Description | +|-------------------------------|-----------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| n | int | 1 | Number of output sequences to return for the given prompt. | +| best_of | Optional[int] | None | Number of output sequences generated from the prompt. The top `n` sequences are returned from these `best_of` sequences. Must be ≥ `n`. Treated as beam width in beam search. Default is `n`. | +| presence_penalty | float | 0.0 | Penalizes new tokens based on their presence in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition. | +| frequency_penalty | float | 0.0 | Penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition. | +| repetition_penalty | float | 1.0 | Penalizes new tokens based on their appearance in the prompt and generated text. Values > 1 encourage new tokens, values < 1 encourage repetition. | +| temperature | float | 1.0 | Controls the randomness of sampling. Lower values make it more deterministic, higher values make it more random. Zero means greedy sampling. | +| top_p | float | 1.0 | Controls the cumulative probability of top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. | +| top_k | int | -1 | Controls the number of top tokens to consider. Set to -1 to consider all tokens. | +| min_p | float | 0.0 | Represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable. | +| use_beam_search | bool | False | Whether to use beam search instead of sampling. | +| length_penalty | float | 1.0 | Penalizes sequences based on their length. Used in beam search. | +| early_stopping | Union[bool, str] | False | Controls stopping condition in beam search. Can be `True`, `False`, or `"never"`. | +| stop | Union[None, str, List[str]] | None | List of strings that stop generation when produced. Output will not contain these strings. | +| stop_token_ids | Optional[List[int]] | None | List of token IDs that stop generation when produced. Output contains these tokens unless they are special tokens. | +| ignore_eos | bool | False | Whether to ignore the End-Of-Sequence token and continue generating tokens after its generation. | +| max_tokens | int | 16 | Maximum number of tokens to generate per output sequence. | +| logprobs | Optional[int] | None | Number of log probabilities to return per output token. | +| prompt_logprobs | Optional[int] | None | Number of log probabilities to return per prompt token. | +| skip_special_tokens | bool | True | Whether to skip special tokens in the output. | +| spaces_between_special_tokens | bool | True | Whether to add spaces between special tokens in the output. | + ## Sample Inputs and Outputs