owen/worker-vllm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Runpod not RunPod

Browse Source
This commit is contained in:
velaraptor-runpod
2026-03-17 20:59:52 -05:00
parent d9808815ee
commit 30e8514d63
2 changed files with 29 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.runpod/README.md
+8 -8
View File
@@ -1,10 +1,10 @@
![vLLM worker banner](https://cpjrphpz3t5wbwfe.public.blob.vercel-storage.com/worker-vllm_banner.jpeg) ![vLLM worker banner](https://image.runpod.ai/preview/vllm/vllm-banner.png)
Run LLMs using [vLLM](https://docs.vllm.ai) with an OpenAI-compatible API Run LLMs using [vLLM](https://docs.vllm.ai) with an OpenAI-compatible API
--- ---
[![RunPod](https://api.runpod.io/badge/runpod-workers/worker-vllm)](https://www.runpod.io/console/hub/runpod-workers/worker-vllm) [![Runpod](https://api.runpod.io/badge/runpod-workers/worker-vllm)](https://www.runpod.io/console/hub/runpod-workers/worker-vllm)
--- ---
@@ -34,11 +34,11 @@ For complete configuration options, see the [full configuration documentation](h
## API Usage ## API Usage
This worker supports two API formats: **RunPod native** and **OpenAI-compatible**. This worker supports two API formats: **Runpod native** and **OpenAI-compatible**.
### RunPod Native API ### Runpod Native API
For testing directly in the RunPod UI, use these examples in your endpoint's request tab. For testing directly in the Runpod UI, use these examples in your endpoint's request tab.
#### Chat Completions #### Chat Completions
@@ -104,7 +104,7 @@ For direct text generation without chat format:
### OpenAI-Compatible API ### OpenAI-Compatible API
For external clients and SDKs, use the `/openai/v1` path prefix with your RunPod API key. For external clients and SDKs, use the `/openai/v1` path prefix with your Runpod API key.
#### Chat Completions #### Chat Completions
@@ -180,7 +180,7 @@ Both APIs return the same response format:
Below are minimal `python` snippets so you can copy-paste to get started quickly. Below are minimal `python` snippets so you can copy-paste to get started quickly.
> Replace `<ENDPOINT_ID>` with your endpoint ID and `<API_KEY>` with a [RunPod API key](https://docs.runpod.io/get-started/api-keys). > Replace `<ENDPOINT_ID>` with your endpoint ID and `<API_KEY>` with a [Runpod API key](https://docs.runpod.io/get-started/api-keys).
### OpenAI compatible API ### OpenAI compatible API
@@ -190,7 +190,7 @@ Minimal Python example using the official `openai` SDK:
from openai import OpenAI from openai import OpenAI
import os import os
# Initialize the OpenAI Client with your RunPod API Key and Endpoint URL # Initialize the OpenAI Client with your Runpod API Key and Endpoint URL
client = OpenAI( client = OpenAI(
api_key=os.getenv("RUNPOD_API_KEY"), api_key=os.getenv("RUNPOD_API_KEY"),
base_url=f"https://api.runpod.ai/v2/<ENDPOINT_ID>/openai/v1", base_url=f"https://api.runpod.ai/v2/<ENDPOINT_ID>/openai/v1",
README.md
+21 -15
View File
@@ -2,10 +2,16 @@
# OpenAI-Compatible vLLM Serverless Endpoint Worker # OpenAI-Compatible vLLM Serverless Endpoint Worker
Deploy OpenAI-Compatible Blazing-Fast LLM Endpoints powered by the [vLLM](https://github.com/vllm-project/vllm) Inference Engine on RunPod Serverless with just a few clicks. Deploy OpenAI-Compatible Blazing-Fast LLM Endpoints powered by the [vLLM](https://github.com/vllm-project/vllm) Inference Engine on Runpod Serverless with just a few clicks.
</div> </div>
![vLLM worker banner](https://image.runpod.ai/preview/vllm/vllm-banner.png)
Current vLLM version: [0.16.0](https://github.com/vllm-project/vllm/releases/tag/v0.16.0)
> Check out our Load Balancer implementation here: [vLLM Load Balancer](https://github.com/runpod-workers/vllm-loadbalancer-ep)
## Table of Contents ## Table of Contents
- [Setting up the Serverless Worker](#setting-up-the-serverless-worker) - [Setting up the Serverless Worker](#setting-up-the-serverless-worker)
@@ -21,7 +27,7 @@ Deploy OpenAI-Compatible Blazing-Fast LLM Endpoints powered by the [vLLM](https:
- [Modifying your OpenAI Codebase to use your deployed vLLM Worker](#modifying-your-openai-codebase-to-use-your-deployed-vllm-worker) - [Modifying your OpenAI Codebase to use your deployed vLLM Worker](#modifying-your-openai-codebase-to-use-your-deployed-vllm-worker)
- [OpenAI Request Input Parameters](#openai-request-input-parameters) - [OpenAI Request Input Parameters](#openai-request-input-parameters)
- [Chat Completions [RECOMMENDED]](#chat-completions-recommended) - [Chat Completions [RECOMMENDED]](#chat-completions-recommended)
- [Examples: Using your RunPod endpoint with OpenAI](#examples-using-your-runpod-endpoint-with-openai) - [Examples: Using your Runpod endpoint with OpenAI](#examples-using-your-runpod-endpoint-with-openai)
- [Chat Completions](#chat-completions) - [Chat Completions](#chat-completions)
- [Getting a list of names for available models](#getting-a-list-of-names-for-available-models) - [Getting a list of names for available models](#getting-a-list-of-names-for-available-models)
- [Usage: Standard (Non-OpenAI)](#usage-standard-non-openai) - [Usage: Standard (Non-OpenAI)](#usage-standard-non-openai)
@@ -33,7 +39,7 @@ Deploy OpenAI-Compatible Blazing-Fast LLM Endpoints powered by the [vLLM](https:
## Option 1: Deploy Any Model Using Pre-Built Docker Image [Recommended] ## Option 1: Deploy Any Model Using Pre-Built Docker Image [Recommended]
**🚀 Deploy Guide**: Follow our [step-by-step deployment guide](https://docs.runpod.io/serverless/vllm/get-started) to deploy using the RunPod Console. **🚀 Deploy Guide**: Follow our [step-by-step deployment guide](https://docs.runpod.io/serverless/vllm/get-started) to deploy using the Runpod Console.
**📦 Docker Image**: `runpod/worker-v1-vllm:<version>` **📦 Docker Image**: `runpod/worker-v1-vllm:<version>`
@@ -148,7 +154,7 @@ The vLLM Worker is fully compatible with OpenAI's API, and you can use it with a
**Python** (similar to Node.js, etc.): **Python** (similar to Node.js, etc.):
1. When initializing the OpenAI Client in your code, change the `api_key` to your RunPod API Key and the `base_url` to your RunPod Serverless Endpoint URL in the following format: `https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1`, filling in your deployed endpoint ID. For example, if your Endpoint ID is `abc1234`, the URL would be `https://api.runpod.ai/v2/abc1234/openai/v1`. 1. When initializing the OpenAI Client in your code, change the `api_key` to your Runpod API Key and the `base_url` to your Runpod Serverless Endpoint URL in the following format: `https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1`, filling in your deployed endpoint ID. For example, if your Endpoint ID is `abc1234`, the URL would be `https://api.runpod.ai/v2/abc1234/openai/v1`.
- Before: - Before:
@@ -174,7 +180,7 @@ The vLLM Worker is fully compatible with OpenAI's API, and you can use it with a
```python ```python
response = client.chat.completions.create( response = client.chat.completions.create(
model="gpt-3.5-turbo", model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
temperature=0, temperature=0,
max_tokens=100, max_tokens=100,
) )
@@ -183,7 +189,7 @@ The vLLM Worker is fully compatible with OpenAI's API, and you can use it with a
```python ```python
response = client.chat.completions.create( response = client.chat.completions.create(
model="<YOUR DEPLOYED MODEL REPO/NAME>", model="<YOUR DEPLOYED MODEL REPO/NAME>",
messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
temperature=0, temperature=0,
max_tokens=100, max_tokens=100,
) )
@@ -191,7 +197,7 @@ The vLLM Worker is fully compatible with OpenAI's API, and you can use it with a
**Using http requests**: **Using http requests**:
1. Change the `Authorization` header to your RunPod API Key and the `url` to your RunPod Serverless Endpoint URL in the following format: `https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1` 1. Change the `Authorization` header to your Runpod API Key and the `url` to your Runpod Serverless Endpoint URL in the following format: `https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1`
- Before: - Before:
```bash ```bash
curl https://api.openai.com/v1/chat/completions \ curl https://api.openai.com/v1/chat/completions \
@@ -202,7 +208,7 @@ The vLLM Worker is fully compatible with OpenAI's API, and you can use it with a
"messages": [ "messages": [
{ {
"role": "user", "role": "user",
"content": "Why is RunPod the best platform?" "content": "Why is Runpod the best platform?"
} }
], ],
"temperature": 0, "temperature": 0,
@@ -219,7 +225,7 @@ The vLLM Worker is fully compatible with OpenAI's API, and you can use it with a
"messages": [ "messages": [
{ {
"role": "user", "role": "user",
"content": "Why is RunPod the best platform?" "content": "Why is Runpod the best platform?"
} }
], ],
"temperature": 0, "temperature": 0,
@@ -239,7 +245,7 @@ When using the chat completion feature of the vLLM Serverless Endpoint Worker, y
| Parameter | Type | Default Value | Description | | Parameter | Type | Default Value | Description |
| ------------------- | -------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ------------------- | -------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `messages` | Union[str, List[Dict[str, str]]] | | List of messages, where each message is a dictionary with a `role` and `content`. The model's chat template will be applied to the messages automatically, so the model must have one or it should be specified as `CUSTOM_CHAT_TEMPLATE` env var. | | `messages` | Union[str, List[Dict[str, str]]] | | List of messages, where each message is a dictionary with a `role` and `content`. The model's chat template will be applied to the messages automatically, so the model must have one or it should be specified as `CUSTOM_CHAT_TEMPLATE` env var. |
| `model` | str | | The model repo that you've deployed on your RunPod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the **Examples: Using your RunPod endpoint with OpenAI** section | | `model` | str | | The model repo that you've deployed on your Runpod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the **Examples: Using your Runpod endpoint with OpenAI** section |
| `temperature` | Optional[float] | 0.7 | Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling. | | `temperature` | Optional[float] | 0.7 | Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling. |
| `top_p` | Optional[float] | 1.0 | Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. | | `top_p` | Optional[float] | 1.0 | Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. |
| `n` | Optional[int] | 1 | Number of output sequences to return for the given prompt. | | `n` | Optional[int] | 1 | Number of output sequences to return for the given prompt. |
@@ -269,15 +275,15 @@ Additional parameters supported by vLLM:
</details> </details>
### Examples: Using your RunPod endpoint with OpenAI ### Examples: Using your Runpod endpoint with OpenAI
First, initialize the OpenAI Client with your RunPod API Key and Endpoint URL: First, initialize the OpenAI Client with your Runpod API Key and Endpoint URL:
```python ```python
from openai import OpenAI from openai import OpenAI
import os import os
# Initialize the OpenAI Client with your RunPod API Key and Endpoint URL # Initialize the OpenAI Client with your Runpod API Key and Endpoint URL
client = OpenAI( client = OpenAI(
api_key=os.environ.get("RUNPOD_API_KEY"), api_key=os.environ.get("RUNPOD_API_KEY"),
base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1", base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
@@ -293,7 +299,7 @@ This is the format used for GPT-4 and focused on instruction-following and chat.
# Create a chat completion stream # Create a chat completion stream
response_stream = client.chat.completions.create( response_stream = client.chat.completions.create(
model="<YOUR DEPLOYED MODEL REPO/NAME>", model="<YOUR DEPLOYED MODEL REPO/NAME>",
messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
temperature=0, temperature=0,
max_tokens=100, max_tokens=100,
stream=True, stream=True,
@@ -307,7 +313,7 @@ This is the format used for GPT-4 and focused on instruction-following and chat.
# Create a chat completion # Create a chat completion
response = client.chat.completions.create( response = client.chat.completions.create(
model="<YOUR DEPLOYED MODEL REPO/NAME>", model="<YOUR DEPLOYED MODEL REPO/NAME>",
messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
temperature=0, temperature=0,
max_tokens=100, max_tokens=100,
) )