velaraptor-runpodandClaude Sonnet 4.6
08580e7ccf
Brings forward the L40 GPU type in tests.json and the new llama/qwen tuned config files, while keeping Dockerfile pinned at vllm 0.20.2 (v2.20.1 state). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
11 lines
308 B
YAML
11 lines
308 B
YAML
model: Qwen/Qwen3-8B
|
|
gpu-memory-utilization: 0.95
|
|
max-model-len: 8192
|
|
dtype: auto
|
|
trust-remote-code: true
|
|
quantization: fp8
|
|
kv-cache-dtype: fp8
|
|
enforce-eager: false
|
|
enable-prefix-caching: true
|
|
speculative-config: '{"model":"RedHatAI/Qwen3-8B-speculator.eagle3","method":"eagle3","num_speculative_tokens":3}'
|