Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 38 additions & 20 deletions docs/commands/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ $ winml build [options]
|---|---|---|---|---|
| `--config` | `-c` | path | `None` | `WinMLBuildConfig` JSON file, generated by `winml config`. If omitted, config is auto-generated from `-m`. |
| `--model` | `-m` | string | `None` | Hugging Face model ID or path to an existing `.onnx` file. |
| `--export-type` | | choice | `generic` | Output selector: `generic` builds the stock single/composite ONNX model; `optimized` builds the family's registered runtime-optimized recipe (today the onnxruntime-genai NPU bundle) for the **resolved** `--ep`/`--device`. `optimized` fails fast if the family has no recipe or the resolved target is not one the recipe supports. |
| `--output-dir` | `-o` | path | `None` | Directory for all build artifacts. Mutually exclusive with `--use-cache`. |
| `--use-cache/--no-use-cache` | | flag | `false` | Store artifacts in the winml-cli global cache (`~/.cache/winml/`). Mutually exclusive with `--output-dir`. |
| `--rebuild/--no-rebuild` | | flag | `false` | Overwrite existing artifacts and re-run the full pipeline. |
Expand Down Expand Up @@ -58,11 +59,10 @@ single-pass build. Individual stages can be suppressed with `--no-quant`,

## Genai bundles for decoder LLMs (NPU + QNN)

For a registered decoder-LLM family (currently **Qwen3**), targeting the NPU HTP
with an explicit `--ep qnn` switches `winml build` to produce a complete
[onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai) **bundle**
directory instead of the stock per-model ONNX output. Pair `--ep qnn` with
`--device npu`, or with `--device auto` when auto-detection resolves to the NPU:
For a registered decoder-LLM family (currently **Qwen3**), `--export-type
optimized` switches `winml build` from the stock per-model ONNX output to a
complete [onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai)
**bundle** directory:

| File | Role | Precision |
|---|---|---|
Expand All @@ -73,20 +73,38 @@ directory instead of the stock per-model ONNX output. Pair `--ep qnn` with
| `genai_config.json` + tokenizer | onnxruntime-genai runtime metadata | — |

```bash
# One command: HF decoder LLM → full onnxruntime-genai bundle on the NPU
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn
# or let device auto-detection pick the NPU:
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device auto --ep qnn
# One command: HF decoder LLM → full onnxruntime-genai bundle
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --export-type optimized
```

The trigger is data-driven and opt-in: **`--ep qnn` must be passed explicitly**,
alongside an NPU target (`--device npu`, or `--device auto` resolving to the NPU).
Every other target — including Qwen3 on CPU/GPU, or an auto-detected NPU target
without an explicit `--ep qnn` — keeps the existing behavior (the stock composite
build). `--output-dir` is required (the bundle is a directory) and `--use-cache`
is not supported. Use `--precision` to override the transformer precision; the CPU
companions always use their bundle-standard precisions. See
[Qwen3 — Genai Bundle](../samples/qwen3-genai-bundle.md) for the full walkthrough.
`--export-type optimized` resolves `--ep`/`--device` the same way a generic
build does — an explicit value is honored, otherwise the host is probed — and
then builds the recipe for that resolved target. On an NPU host the command
above needs no flags; on a host without the accelerator the resolved target has
no recipe and the build fails fast (naming the `ep`/`device`) instead of
silently producing a generic model. Pin `--ep qnn --device npu` to build the
bundle anywhere (e.g. CI), since an explicit target skips host detection. An
unregistered family, a pre-exported `.onnx` input, or module mode also fail fast.
`--output-dir` is required (the bundle is a directory) and `--use-cache` is not
supported. Use `--precision` to override the transformer precision; the CPU
companions always use their bundle-standard precisions.

!!! note "Backward-compatible shortcut"
When `--export-type` is omitted, a registered family with an explicit
`--ep qnn` on an NPU target still routes to its optimized bundle. The NPU
target may be explicit (`--device npu`) or resolved from `auto` — whether
`--device auto` is typed or `--device` is left off entirely:

```bash
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn
# or let device auto-detection pick the NPU (--device may be omitted):
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --ep qnn
```

`--export-type generic` always forces the stock composite build, even for a
registered family on the NPU.

See [Qwen3 — Genai Bundle](../samples/qwen3-genai-bundle.md) for the full walkthrough.

## Examples

Expand Down Expand Up @@ -150,9 +168,9 @@ winml build -m microsoft/resnet-50 -o output/ \
pre-exported ONNX inputs because the export step has already happened.
- **Existing artifacts are reused by default.** Pass `--rebuild` to force a
fresh run after changing the config.
- **Genai bundles require `--output-dir`, not `--use-cache`.** A decoder-LLM
bundle (`--device npu --ep qnn`) writes a directory of components and rejects
`--use-cache`.
- **Genai bundles require `--output-dir`, not `--use-cache`.** An optimized
decoder-LLM bundle (`--export-type optimized`, or the `--device npu --ep qnn`
shortcut) writes a directory of components and rejects `--use-cache`.

## See also

Expand Down
39 changes: 26 additions & 13 deletions docs/samples/qwen3-genai-bundle.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ memory-bound companions stay on CPU.

```mermaid
graph LR
A["winml build -m Qwen/Qwen3-0.6B --device npu --ep qnn"] --> B[Genai bundle recipe]
A["winml build -m Qwen/Qwen3-0.6B --export-type optimized"] --> B[Genai bundle recipe]
B --> C[ctx.onnx / iter.onnx — NPU]
B --> D[embeddings.onnx — CPU]
B --> E[lm_head.onnx — CPU]
Expand All @@ -41,20 +41,21 @@ graph LR

## Step 1: Build the bundle (one command)

Targeting the NPU HTP with an explicit `--ep qnn` switches `winml build` from the
stock per-model ONNX output to the full genai bundle. Pair it with `--device npu`,
or `--device auto` when auto-detection resolves to the NPU:
`--export-type optimized` switches `winml build` from the stock per-model ONNX
output to the full genai bundle. It resolves `--ep`/`--device` like a normal
build — honoring an explicit value, otherwise probing the host — and builds the
recipe for that resolved target, so on an NPU host no flags are needed:

```bash
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn
# or, letting device auto-detection pick the NPU:
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device auto --ep qnn
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --export-type optimized
```

This builds (or reuses from cache) all four components and assembles them, writing
`out/qwen3-bundle/genai_config.json` alongside the ONNX graphs and tokenizer.
`--output-dir` is required — the bundle is a directory — and `--use-cache` is not
supported for bundles.
supported for bundles. On a host without the NPU the resolved target has no recipe
and the build fails fast; pin `--ep qnn --device npu` to build the bundle anywhere
(e.g. CI), since an explicit target skips host detection.

The Qwen3 transformer's quantization scheme is fixed by its recipe (`w8a16`, the
scheme its QNN HTP export is tuned for), so it is not overridable — passing a
Expand All @@ -63,11 +64,23 @@ The CPU companions likewise keep their bundle-standard precisions.

Force a clean rebuild of every component with `--rebuild`.

!!! note "Opt-in, non-destructive"
The bundle path only triggers on an **explicit `--ep qnn`** together with an
NPU target — either `--device npu` or a `--device auto` that resolves to the
NPU. Qwen3 on any other target — CPU, GPU, or an auto-detected NPU *without*
an explicit `--ep qnn` — still produces the stock composite build, unchanged.
!!! note "Backward-compatible shortcut"
When `--export-type` is omitted, an **explicit `--ep qnn`** targeting the NPU
still routes a registered family to its bundle. The NPU target may be
explicit (`--device npu`) or resolved from `auto` — whether `--device auto`
is typed or `--device` is omitted:

```bash
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn
# or, letting device auto-detection pick the NPU (--device may be omitted):
winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --ep qnn
```

Qwen3 on any other target — CPU, GPU, or an auto-detected NPU *without* an
explicit `--ep qnn` — still produces the stock composite build. `--export-type
generic` always forces the stock build, even on the NPU. A pinned
`--ep`/`--device` that contradicts the recipe fails fast rather than silently
reverting.

## Step 2: Tune context and prefill lengths (optional)

Expand Down
Loading
Loading