[andweng]

Problem

On a Kubernetes node with RTX PRO 6000 Blackwell GPUs, the kubernetes backend offers no GPUs. NVIDIA GPU Feature Discovery (k8s-device-plugin v0.20.1) labels the node:

nvidia.com/gpu.product=NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition
nvidia.com/gpu.count=2

and nvidia.com/gpu: 2 is allocatable, but:

$ dstack offer --gpu 1..
...
No matching instance offers available. Possible reasons:
https://dstack.ai/docs/guides/troubleshooting/#no-offers

dstack offer without --gpu lists the node as cpu=x86:1 only.

In get_nvidia_gpu_from_node_labels, replace("RTX-", "RTX") turns the label into NVIDIA-RTXPRO-6000-Blackwell-..., whose tokens are rtxpro, 6000, ... None equals the known name RTXPRO6000, so the function returns None:

get_nvidia_gpu_from_node_labels(
    {"nvidia.com/gpu.product": "NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition"}
)
# None

Reproduced with 0.22.8 against a single-node RKE2 v1.36.6 cluster; same code on master.

Fix

Replace RTX-PRO- with RTXPRO before the existing RTX- → RTX step, so the label has the token rtxpro6000. Other products are unaffected: only labels containing RTX-PRO- change.

With the change applied to 0.22.6 on the same cluster, dstack offer --gpu 1.. lists gpu=RTXPRO6000:96GB:1.

Tests

test_returns_rtx_gpu in src/tests/_internal/core/backends/kubernetes/test_resources.py: RTX PRO 6000 Max-Q (the label from the node above), RTX PRO 6000 Server Edition, RTX PRO 4500, and GeForce RTX 4090 (unchanged behaviour). The three RTX PRO cases fail without the fix.

• uv run ruff check --fix, uv run ruff format: no changes
• uv run pytest src/tests: 5129 passed, 1543 skipped