owen/worker-vllm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #226 from runpod-workers/fix/cse-839-max-concurrency
Release / release (push) Waiting to run

Browse Source 
fix: max concurrency = 30 instead of 300
This commit is contained in:
Tim Pietrusky
2025-10-22 22:54:21 +02:00
committed by GitHub
parent 60c8f257a8
commit 66e1b1605b
6 changed files with 7 additions and 1520 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/conventions.md
+3 -2
View File
@@ -51,7 +51,8 @@ RunPod Request → handler.py → JobInput → Engine Selection → vLLM Generat
- `src/engine_args.py`: Centralized configuration management
- `src/constants.py`: Default values for core settings
- `worker-config.json`: UI form generation for RunPod console
- `.runpod/hub.json`: Hub UI configuration (CRITICAL: always update when changing defaults)
- `worker-config.json`: UI form generation for RunPod console (if exists)
## Core Development Concepts
@@ -222,7 +223,7 @@ src/
### 2. **Concurrency Patterns**
- **Max Concurrency**: 300 concurrent requests by default
- **Max Concurrency**: 30 concurrent requests by default
- **vLLM Queuing**: Internal request batching and scheduling
- **RunPod Integration**: Concurrency modifier for auto-scaling