worker-vllm/docs/configuration.md

Configuration Reference

Complete guide to all environment variables and configuration options for worker-vllm.

LLM Settings

Variable	Default	Type/Choices	Description
MODEL_NAME	'facebook/opt-125m'	str	Name or path of the Hugging Face model to use.
MODEL_REVISION	'main'	str	Model revision to load (default: main).
TOKENIZER	None	str	Name or path of the Hugging Face tokenizer to use.
SKIP_TOKENIZER_INIT	False	bool	Skip initialization of tokenizer and detokenizer.
TOKENIZER_MODE	'auto'	['auto', 'slow']	The tokenizer mode.
TRUST_REMOTE_CODE	False	bool	Trust remote code from Hugging Face.
DOWNLOAD_DIR	None	str	Directory to download and load the weights.
LOAD_FORMAT	'auto'	str	The format of the model weights to load.
HF_TOKEN	-	str	Hugging Face token for private and gated models.
DTYPE	'auto'	['auto', 'half', 'float16', 'bfloat16', 'float', 'float32']	Data type for model weights and activations.
KV_CACHE_DTYPE	'auto'	['auto', 'fp8']	Data type for KV cache storage.
MAX_MODEL_LEN	None	int	Model context length.
DISTRIBUTED_EXECUTOR_BACKEND	None	['ray', 'mp']	Backend to use for distributed serving.
PIPELINE_PARALLEL_SIZE	1	int	Number of pipeline stages.
TENSOR_PARALLEL_SIZE	1	int	Number of tensor parallel replicas.
MAX_PARALLEL_LOADING_WORKERS	None	int	Load model sequentially in multiple batches.
RAY_WORKERS_USE_NSIGHT	False	bool	If specified, use nsight to profile Ray workers.
ENABLE_PREFIX_CACHING	False	bool	Enables automatic prefix caching.
DISABLE_SLIDING_WINDOW	False	bool	Disables sliding window, capping to sliding window size.
SEED	0	int	Random seed for operations.
NUM_GPU_BLOCKS_OVERRIDE	None	int	If specified, ignore GPU profiling result and use this number of GPU blocks.
MAX_NUM_BATCHED_TOKENS	None	int	Maximum number of batched tokens per iteration.
MAX_NUM_SEQS	256	int	Maximum number of sequences per iteration.
MAX_LOGPROBS	20	int	Max number of log probs to return when logprobs is specified in SamplingParams.
DISABLE_LOG_STATS	False	bool	Disable logging statistics.
QUANTIZATION	None	['awq', 'squeezellm', 'gptq', 'bitsandbytes']	Method used to quantize the weights.

LoRA (Low-Rank Adaptation) Settings

Variable	Default	Type	Description
ENABLE_LORA	False	bool	If True, enable handling of LoRA adapters.
MAX_LORAS	1	int	Max number of LoRAs in a single batch.
MAX_LORA_RANK	16	int	Max LoRA rank.
LORA_DTYPE	'auto'	['auto', 'float16', 'bfloat16', 'float32']	Data type for LoRA.
MAX_CPU_LORAS	None	int	Maximum number of LoRAs to store in CPU memory.
FULLY_SHARDED_LORAS	False	bool	Enable fully sharded LoRA layers.
LORA_MODULES	[]	list[dict]	Add lora adapters from Hugging Face [{"name": "xx", "path": "xxx/xxxx", "base_model_name": "xxx/xxxx"}]

Note: When using LoRA with serverless deployments, the OpenAI serving engines are initialized on the first request (deferred initialization) to avoid event loop conflicts. LoRA adapter count is logged at startup.

Speculative Decoding Settings

Speculative decoding can be configured in two ways:

Option 1: JSON Configuration

Set SPECULATIVE_CONFIG to a JSON string with your full speculative decoding configuration:

SPECULATIVE_CONFIG='{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}'

Option 2: Individual Environment Variables

Variable	Default	Type/Choices	Description
SPECULATIVE_METHOD	None	['draft_model', 'ngram', 'eagle', 'eagle3', 'medusa', 'mlp_speculator']	Speculative decoding method to use.
SPECULATIVE_MODEL	None	str	The name of the draft model to be used in speculative decoding.
NUM_SPECULATIVE_TOKENS	None	int	The number of speculative tokens to sample from the draft model.
SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE	None	int	Number of tensor parallel replicas for the draft model.
SPECULATIVE_MAX_MODEL_LEN	None	int	The maximum sequence length supported by the draft model.
SPECULATIVE_DISABLE_BY_BATCH_SIZE	None	int	Disable speculative decoding if the number of enqueue requests is larger than this value.
NGRAM_PROMPT_LOOKUP_MAX	None	int	Max size of window for ngram prompt lookup in speculative decoding.
NGRAM_PROMPT_LOOKUP_MIN	None	int	Min size of window for ngram prompt lookup in speculative decoding.

If SPECULATIVE_CONFIG is set, it takes priority over individual env vars. When using individual env vars without SPECULATIVE_METHOD, the method is auto-detected from the model name or configuration.

Scheduling & Performance Settings

Variable	Default	Type/Choices	Description
GPU_MEMORY_UTILIZATION	0.95	float	Sets GPU VRAM utilization.
MAX_PARALLEL_LOADING_WORKERS	None	int	Load model sequentially in multiple batches, to avoid RAM OOM when using tensor parallel and large models.
BLOCK_SIZE	16	8, 16, 32	Token block size for contiguous chunks of tokens.
SWAP_SPACE	4	int	CPU swap space size (GiB) per GPU.
ENFORCE_EAGER	False	bool	Always use eager-mode PyTorch. If False(0), will use eager mode and CUDA graph in hybrid for maximal performance and flexibility.
MAX_SEQ_LEN_TO_CAPTURE	8192	int	Maximum context length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode.
DISABLE_CUSTOM_ALL_REDUCE	0	int	Enables or disables custom all reduce.
ENABLE_EXPERT_PARALLEL	False	bool	Enable Expert Parallel for MoE models.
SCHEDULER_DELAY_FACTOR	0.0	float	Apply a delay before scheduling next prompt.
ENABLE_CHUNKED_PREFILL	False	bool	Enable chunked prefill requests.
ATTENTION_BACKEND	None	str	Attention backend to use (e.g., FLASH_ATTN, XFORMERS, FLASHINFER).
ASYNC_SCHEDULING	False	bool	Enable async scheduling for improved throughput.
STREAM_INTERVAL	0	float	Interval in seconds between streaming responses.

Tokenizer Settings

Variable	Default	Type/Choices	Description
TOKENIZER_NAME	None	str	Tokenizer repository to use a different tokenizer than the model's default.
TOKENIZER_REVISION	None	str	Tokenizer revision to load.
CUSTOM_CHAT_TEMPLATE	None	str of single-line jinja template	Custom chat jinja template. More Info

Streaming & Batch Settings

The way this works is that the first request will have a batch size of DEFAULT_MIN_BATCH_SIZE, and each subsequent request will have a batch size of previous_batch_size * DEFAULT_BATCH_SIZE_GROWTH_FACTOR. This will continue until the batch size reaches DEFAULT_BATCH_SIZE. E.g. for the default values, the batch sizes will be 1, 3, 9, 27, 50, 50, 50, .... You can also specify this per request, with inputs max_batch_size, min_batch_size, and batch_size_growth_factor. This has nothing to do with vLLM's internal batching, but rather the number of tokens sent in each HTTP request from the worker.

Variable	Default	Type/Choices	Description
DEFAULT_BATCH_SIZE	50	int	Default and Maximum batch size for token streaming to reduce HTTP calls.
DEFAULT_MIN_BATCH_SIZE	1	int	Batch size for the first request, which will be multiplied by the growth factor every subsequent request.
DEFAULT_BATCH_SIZE_GROWTH_FACTOR	3	float	Growth factor for dynamic batch size.

OpenAI Compatibility Settings

Variable	Default	Type/Choices	Description
RAW_OPENAI_OUTPUT	1	boolean as int	Enables raw OpenAI SSE format string output when streaming. Required to be enabled (which it is by default) for OpenAI compatibility.
OPENAI_SERVED_MODEL_NAME_OVERRIDE	None	str	Overrides the name of the served model from model repo/path to specified name, which you will then be able to use the value for the model parameter when making OpenAI requests
OPENAI_RESPONSE_ROLE	assistant	str	Role of the LLM's Response in OpenAI Chat Completions.
ENABLE_AUTO_TOOL_CHOICE	false	bool	Enables automatic tool selection for supported models. Set to true to activate.
TOOL_CALL_PARSER	None	str	Specifies the parser for tool calls. Options: mistral, hermes, llama3_json, llama4_json, llama4_pythonic, granite, granite-20b-fc, deepseek_v3, internlm, jamba, phi4_mini_json, pythonic
REASONING_PARSER	None	str	Parser for reasoning-capable models (enables reasoning mode). Examples: deepseek_r1, qwen3, granite, hunyuan_a13b. Leave unset to disable.
TRUST_REQUEST_CHAT_TEMPLATE	false	bool	Allow chat templates from incoming requests to override the server default.
RETURN_TOKENS_AS_TOKEN_IDS	false	bool	Return token IDs instead of token strings in responses.
EXCLUDE_TOOLS_WHEN_TOOL_CHOICE_NONE	false	bool	When tool_choice is 'none', exclude tool definitions from the prompt.
ENABLE_PROMPT_TOKENS_DETAILS	false	bool	Enable detailed prompt token usage in responses.
ENABLE_FORCE_INCLUDE_USAGE	false	bool	Force include usage information in all streaming responses.
ENABLE_LOG_OUTPUTS	false	bool	Log model outputs for debugging.
LOG_ERROR_STACK	false	bool	Log full error stack traces.

Serverless & Concurrency Settings

Variable	Default	Type/Choices	Description
MAX_CONCURRENCY	30	int	Max concurrent requests per worker. vLLM has an internal queue, so you don't have to worry about limiting by VRAM, this is for improving scaling/load balancing efficiency
DISABLE_LOG_STATS	False	bool	Enables or disables vLLM stats logging.
ENABLE_LOG_REQUESTS	False	bool	Enables vLLM request logging.

Docker Build Arguments

These variables are used when building custom Docker images with models baked in:

Variable	Default	Type	Description
BASE_PATH	/runpod-volume	str	Storage directory for huggingface cache and model
WORKER_CUDA_VERSION	12.9.1	str	CUDA version for the worker image

Deprecated Variables

The following variables are deprecated and will be removed in future versions:

Old Variable	New Variable / Migration	Note
MAX_CONTEXT_LEN_TO_CAPTURE	MAX_SEQ_LEN_TO_CAPTURE	Use new variable name
kv_cache_dtype=fp8_e5m2	kv_cache_dtype=fp8	Simplified fp8 format
DISABLE_LOG_REQUESTS	ENABLE_LOG_REQUESTS	Logic inverted: DISABLE_LOG_REQUESTS=true → ENABLE_LOG_REQUESTS=false
VLLM_ATTENTION_BACKEND	ATTENTION_BACKEND	Use new variable name
WORKER_USE_RAY	DISTRIBUTED_EXECUTOR_BACKEND=ray	Removed in vLLM 0.15.0
USE_V2_BLOCK_MANAGER	(removed)	V2 block manager is now the default
NUM_LOOKAHEAD_SLOTS	(removed)	No longer a separate config
ROPE_SCALING	(removed)	Use model config directly
ROPE_THETA	(removed)	Use model config directly
TOKENIZER_POOL_SIZE	(removed)	Removed in vLLM 0.15.0
TOKENIZER_POOL_TYPE	(removed)	Removed in vLLM 0.15.0
TOKENIZER_POOL_EXTRA_CONFIG	(removed)	Removed in vLLM 0.15.0
QUANTIZATION_PARAM_PATH	(removed)	Removed in vLLM 0.15.0
LORA_EXTRA_VOCAB_SIZE	(removed)	Removed in vLLM 0.15.0
LONG_LORA_SCALING_FACTORS	(removed)	Removed in vLLM 0.15.0
GUIDED_DECODING_BACKEND	(removed)	Removed in vLLM 0.15.0
SPEC_DECODING_ACCEPTANCE_METHOD	SPECULATIVE_CONFIG (JSON)	Use JSON config
TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_THRESHOLD	SPECULATIVE_CONFIG (JSON)	Use JSON config
TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_ALPHA	SPECULATIVE_CONFIG (JSON)	Use JSON config
SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE	SPECULATIVE_CONFIG (JSON) or individual env	Still supported via env var
PREEMPTION_MODE	(removed)	Removed in vLLM 0.15.0
PREEMPTION_CHECK_PERIOD	(removed)	Removed in vLLM 0.15.0
PREEMPTION_CPU_CAPACITY	(removed)	Removed in vLLM 0.15.0
MAX_LOG_LEN	(removed)	Removed in vLLM 0.15.0
DISABLE_LOGGING_REQUEST	ENABLE_LOG_REQUESTS	Removed in vLLM 0.15.0
MAX_SEQ_LEN_TO_CAPTURE	(removed)	Removed in vLLM 0.15.0