Files
worker-vllm/src/constants.py
T
2024-03-06 17:08:57 +00:00

4 lines
117 B
Python

DEFAULT_BATCH_SIZE = 50
DEFAULT_MAX_CONCURRENCY = 300
DEFAULT_BATCH_SIZE_GROWTH_FACTOR = 3
DEFAULT_MIN_BATCH_SIZE = 1