add ENABLE_EXPERT_PARALLEL engine arg for MoE models (#239 )

* enable expert parallel arg for moe models * add ENABLE_EXPERT_PARALLEL to hub config
feat: bump transformers to allow Qwen3-VL (#225 )
2025-11-17 19:25:19 +01:00 · 2025-11-14 17:23:34 +01:00 · 2025-11-14 17:23:09 +01:00 · 2025-11-14 17:22:43 +01:00 · 2025-10-31 13:32:34 -04:00 · 2025-10-29 13:06:33 -04:00
5 changed files with 17 additions and 23 deletions
@@ -6,20 +6,10 @@
  "iconUrl": "https://registry.npmmirror.com/@lobehub/icons-static-png/latest/files/dark/vllm-color.png",
  "config": {
    "runsOn": "GPU",
-    "containerDiskInGb": 200,
+    "containerDiskInGb": 150,
    "gpuIds": "ADA_80_PRO,AMPERE_80",
    "gpuCount": 1,
-    "allowedCudaVersions": [
-      "12.9",
-      "12.8",
-      "12.7",
-      "12.6",
-      "12.5",
-      "12.4",
-      "12.3",
-      "12.2",
-      "12.1"
-    ],
+    "allowedCudaVersions": ["12.9", "12.8", "12.7", "12.6", "12.5", "12.4"],
    "presets": [
      {
        "name": "deepseek-ai/deepseek-r1-distill-llama-8b",
@@ -939,6 +929,16 @@
          "advanced": true
        }
      },
+      {
+        "key": "ENABLE_EXPERT_PARALLEL",
+        "input": {
+          "name": "Enable Expert Parallel",
+          "type": "boolean",
+          "description": "Enable Expert Parallel for MoE models",
+          "default": false,
+          "advanced": true
+        }
+      },
      {
        "key": "MODEL_REVISION",
        "input": {
@@ -38,14 +38,6 @@
        "value": "HuggingFaceTB/SmolLM2-135M-Instruct"
      }
    ],
-    "allowedCudaVersions": [
-      "12.7",
-      "12.6",
-      "12.5",
-      "12.4",
-      "12.3",
-      "12.2",
-      "12.1"
-    ]
+    "allowedCudaVersions": ["12.9", "12.8", "12.7", "12.6", "12.5"]
  }
 }
@@ -8,7 +8,7 @@ typing-extensions>=4.8.0
 pydantic
 pydantic-settings
 hf-transfer
-transformers>=4.55.0
+transformers>=4.57.0
 bitsandbytes>=0.45.0
 kernels
 torch==2.6.0
@@ -85,6 +85,7 @@ Complete guide to all environment variables and configuration options for worker
 | `ENFORCE_EAGER`                | False   | `bool`          | Always use eager-mode PyTorch. If False(`0`), will use eager mode and CUDA graph in hybrid for maximal performance and flexibility. |
 | `MAX_SEQ_LEN_TO_CAPTURE`       | `8192`  | `int`           | Maximum context length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode.     |
 | `DISABLE_CUSTOM_ALL_REDUCE`    | `0`     | `int`           | Enables or disables custom all reduce.                                                                                              |
+| `ENABLE_EXPERT_PARALLEL`       | `False` | `bool`           |  Enable Expert Parallel for MoE models  |

 ## Tokenizer Settings

@@ -80,6 +80,7 @@ DEFAULT_ARGS = {
    "guided_decoding_backend": os.getenv('GUIDED_DECODING_BACKEND', 'outlines'),
    "speculative_model": os.getenv('SPECULATIVE_MODEL', None),
    "speculative_draft_tensor_parallel_size": int(os.getenv('SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE', 0)) or None,
+    "enable_expert_parallel": bool(os.getenv('ENABLE_EXPERT_PARALLEL', 'False').lower() == 'true'),
    "num_speculative_tokens": int(os.getenv('NUM_SPECULATIVE_TOKENS', 0)) or None,
    "speculative_max_model_len": int(os.getenv('SPECULATIVE_MAX_MODEL_LEN', 0)) or None,
    "speculative_disable_by_batch_size": int(os.getenv('SPECULATIVE_DISABLE_BY_BATCH_SIZE', 0)) or None,
Author	SHA1	Message	Date
chrisvelaandGitHub	3851d53f93	add ENABLE_EXPERT_PARALLEL engine arg for MoE models (#239 ) Release / release (push) Waiting to run * enable expert parallel arg for moe models * add ENABLE_EXPERT_PARALLEL to hub config	2025-11-17 19:25:19 +01:00
Witold WydmańskiandGitHub	c896438f21	feat: bump transformers to allow Qwen3-VL (#225 ) Release / release (push) Waiting to run	2025-11-14 17:23:34 +01:00
Tim PietruskyandGitHub	912892f94e	fix: remove space from gpuIds (#234 )	2025-11-14 17:23:09 +01:00
Tim PietruskyandGitHub	f8bf82469c	fix(config): update allowed cuda versions in hub and tests config (#236 ) remove unsupported cuda versions (12.1-12.3) from hub.json and tests.json to fix compatibility issues with worker deployment - hub.json: remove 12.1, 12.2, 12.3 from allowedCudaVersions - tests.json: remove 12.1, 12.2, 12.3, 12.4 from allowedCudaVersions refs: AE-1452	2025-11-14 17:22:43 +01:00
Hailong YangandGitHub	ec1664902b	Merge pull request #230 from runpod-workers/feat/cse-853-vllm-template-params Feat/cse 853 vllm template params	2025-10-31 13:32:34 -04:00
Eugene Klitenik	d09122de4a	remove un-needed	2025-10-29 13:06:33 -04:00
Eugene Klitenik	e27dc68dea	remove uneeded	2025-10-29 13:05:26 -04:00
Eugene Klitenik	1ee18d06a9	determine num gpus in python	2025-10-29 11:31:42 -04:00
Eugene Klitenik	5c4edd15cc	update entrypoint command	2025-10-28 18:07:18 -04:00
Eugene Klitenik	b074d3a23b	auto detect num GPUs	2025-10-28 14:39:55 -04:00
Eugene Klitenik	205847471c	reduce default container disk size to 150GB	2025-10-28 13:38:16 -04:00
Tim PietruskyandGitHub	6337a6673a	fix: allow also CUDA 12.8 & 12.9 (#228 ) Release / release (push) Waiting to run	2025-10-24 18:48:26 +02:00