Skip to content

Commit a2eeaec

Browse files
un-defclaude
andauthored
[Docs] Bump Dynamo to 1.6.2 and add retry to Dynamo examples (#4485)
In Dynamo 1.5.1, a vLLM prefill worker that loses etcd hangs on exit: its teardown never shuts down the EngineCore child process, so Python blocks joining it (ai-dynamo/dynamo#16401, fixed in 1.6.0). On 1.6.2, the frontend and both workers exit once their etcd leases expire. Since #4474, services with the `dynamo` router accept `retry`. Without it, a failed replica, such as etcd, fails the whole run. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 89e2028 commit a2eeaec

2 files changed

Lines changed: 20 additions & 8 deletions

File tree

‎mkdocs/docs/concepts/services.md‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,7 @@ Below are examples for running `Qwen/Qwen3.8-Flash-Next` with SMG and `zai-org/G
261261
- name: frontend
262262
replicas: 1
263263
# The vLLM image provides vLLM's GLM tool call and reasoning parsers
264-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
264+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
265265
commands:
266266
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
267267
- |
@@ -279,7 +279,7 @@ Below are examples for running `Qwen/Qwen3.8-Flash-Next` with SMG and `zai-org/G
279279

280280
- name: worker
281281
replicas: 2
282-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
282+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
283283
commands:
284284
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
285285
# vLLM binds the KV events endpoint only if it's a wildcard address, otherwise it connects
@@ -294,6 +294,10 @@ Below are examples for running `Qwen/Qwen3.8-Flash-Next` with SMG and `zai-org/G
294294

295295
port: 8000
296296
model: zai-org/GLM-4.5-Air-FP8
297+
298+
# Resubmit a failed replica, such as etcd, instead of failing the whole run
299+
retry:
300+
on_events: [error, interruption]
297301
```
298302

299303
</div>
@@ -421,7 +425,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
421425
- name: frontend
422426
replicas: 1
423427
# The vLLM image provides vLLM's GLM tool call and reasoning parsers
424-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
428+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
425429
commands:
426430
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
427431
- |
@@ -440,7 +444,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
440444
# Only prefill workers publish KV events, as Dynamo's decode routing ignores prefix overlap
441445
- name: prefill-worker
442446
replicas: 4
443-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
447+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
444448
commands:
445449
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
446450
- export VLLM_NIXL_SIDE_CHANNEL_HOST=$DSTACK_MASTER_NODE_IP
@@ -458,7 +462,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
458462

459463
- name: decode-worker
460464
replicas: 8
461-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
465+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
462466
commands:
463467
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
464468
- export VLLM_NIXL_SIDE_CHANNEL_HOST=$DSTACK_MASTER_NODE_IP
@@ -474,6 +478,10 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
474478

475479
port: 8000
476480
model: zai-org/GLM-4.5-Air-FP8
481+
482+
# Resubmit a failed replica, such as etcd, instead of failing the whole run
483+
retry:
484+
on_events: [error, interruption]
477485
```
478486

479487
</div>

‎mkdocs/docs/examples/inference/dynamo.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ groups:
4343
- name: frontend
4444
replicas: 1
4545
# The vLLM image provides vLLM's GLM tool call and reasoning parsers
46-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
46+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
4747
commands:
4848
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
4949
- |
@@ -65,7 +65,7 @@ groups:
6565
scaling:
6666
metric: rps
6767
target: 3
68-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
68+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
6969
commands:
7070
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
7171
- export VLLM_NIXL_SIDE_CHANNEL_HOST=$DSTACK_MASTER_NODE_IP
@@ -86,7 +86,7 @@ groups:
8686
scaling:
8787
metric: rps
8888
target: 2
89-
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
89+
image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.2
9090
commands:
9191
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379
9292
- export VLLM_NIXL_SIDE_CHANNEL_HOST=$DSTACK_MASTER_NODE_IP
@@ -102,6 +102,10 @@ groups:
102102

103103
port: 8000
104104
model: zai-org/GLM-4.5-Air-FP8
105+
106+
# Resubmit a failed replica, such as etcd, instead of failing the whole run
107+
retry:
108+
on_events: [error, interruption]
105109
```
106110
107111
</div>

0 commit comments

Comments
 (0)