diff --git a/README.md b/README.md index 5042b9a..9c7d34a 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,7 @@ Ensure that you have Docker installed and properly set up before running the doc ## Model Inputs +You may either use a `prompt` or a list of `messages` as input. If you use `messages`, the model's chat template will be applied to the messages automatically, so the model must have one. If you use `prompt`, you may optionally apply the model's chat template to the prompt by setting `apply_chat_template` to `true`. | Argument | Type | Default | Description | |-----------------|------|--------------------|-----------------------------------------------------------------------------------------------| | `prompt` | str | | Prompt string to generate text based on. | @@ -81,6 +82,32 @@ Ensure that you have Docker installed and properly set up before running the doc | `stream` | bool | False | Whether to enable streaming of output. If True, responses are streamed as they are generated. | | `batch_size` | int | DEFAULT_BATCH_SIZE | The number of responses to generate in one batch. Only applicable | +### Messages Format +Your list can contain any number of messages, and each message can have any role from the following list: +- `user` +- `assistant` +- `system` + +The model's chat template will be applied to the messages automatically. + +Example: +```json +[ + { + "role": "system", + "content": "..." + }, + { + "role": "user", + "content": "..." + }, + { + "role": "assistant", + "content": "..." + } +] +``` + ### Sampling Parameters | Argument | Type | Default | Description | |-------------------------------|-----------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -123,8 +150,8 @@ Functions like a text completion model. If the model tokenizer does not have a c #### Output: ```json { - "delayTime": ..., - "executionTime": ..., + "delayTime": 1234, + "executionTime": 1234, "id": "...", "output": [ [ @@ -157,8 +184,8 @@ Functions like a Chat model #### Output: ```json { - "delayTime": ..., - "executionTime": ..., + "delayTime": 1234, + "executionTime": 1234, "id": "...", "output": [ [ @@ -175,6 +202,53 @@ Functions like a Chat model } ``` +### List of Messages (Chat Template applied by default), No Streaming +Functions like a Chat model with a list of messages, to which the model's chat template is applied. You may also use a "system" role and message. +#### Input: +```json +{ + "input": { + "messages": [ + { + "role": "user", + "content": "Tell me why RunPod is the best GPU provider" + }, + { + "role": "assistant", + "content": "RunPod is the best GPU provider for several reasons." + }, + { + "role": "user", + "content": "Name 3 resons" + } + ], + "sampling_params": { + "max_tokens": 100 + } + } +} +``` +#### Output: +```json +{ + "delayTime": 1234, + "executionTime": 1234, + "id": "...", + "output": [ + [ + { + "text": " 1. Cutting-edge hardware: RunPod offers state-of-the-art GPUs from industry-leading manufacturers, ensuring that users have access to the latest technology for their GPU needs.\n\n2. Scalability and flexibility: RunPod provides a wide range of GPU options, allowing users to easily scale their resources up or down depending on their specific requirements, and pay only for what they use.\n\n3. Exceptional customer support: RunPod is dedicated to providing outstanding", + "usage": { + "input": 59, + "output": 100 + } + } + ] + ], + "status": "COMPLETED" +} +``` + ### Chat Template, Streaming Functions like a Chat model, but with streaming output. This is the recommended way to use the vLLM worker. #### Input: @@ -194,8 +268,8 @@ Functions like a Chat model, but with streaming output. This is the recommended #### Output: ```json { - "delayTime": ..., - "executionTime": ..., + "delayTime": 1234, + "executionTime": 1234, "id": "...", "output": [ [