[ExecuTorch][WebGPU] Test coverage for the f16 KV cache#20820
Merged
Conversation
Pull Request resolved: #20773 Tests for the opt-in f16 KV cache (stacked op diff). When built with `EXECUTORCH_WEBGPU_KV_F16` and run on a shader-f16 device, the SDPA op stores the K/V cache as f16, so the entire existing `sdpa_with_kv_cache` golden suite (config sweep + replay + dynamic decode) exercises the f16-KV path with no new goldens needed; this diff loosens the SDPA numeric tolerance to the f16 read-precision floor when — and only when — that path is active. Key changes: - `test_webgpu_native.cpp` `sdpa_within_tol` — under `#ifdef WGPU_BACKEND_KV_F16`, compare at abs `2e-3` / rel `1e-2` when the device negotiated shader-f16, else keep the strict f32 abs `1e-4` / rel `1e-3`. Every SDPA config/replay/decode golden then validates the f16-KV output through the existing exemplars. Constraints: the default (flag-OFF) test build is unchanged; on a non-shader-f16 device (including the CI software adapter) the f16 KV path stays inactive and the strict f32 tolerance applies, so there is no CI behavior change. The f16-KV numeric validation is therefore shader-f16-device-only (Canary/Metal), matching the op's opt-in gating; no CI script change (mirrors the steel-f16 tests). Co-authored-with: Claude Code. ghstack-source-id: 401515179 @exported-using-ghexport Differential Revision: [D110919973](https://our.internmc.facebook.com/intern/diff/D110919973/)
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20820
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Pull Request resolved: #20797 Problem: `WebGPUGraph::create_scratch_buffer` keeps every fused-op scratch `WGPUBuffer` alive in `scratch_buffers_` until graph teardown. The SDPA prefill workspace (`attn_weights` + `attn_weights_softmax`, each `Hq*S*Cmax*4` bytes) and the FlashDecoding partials (`part_o`/`part_ml`) are allocated fresh in every decoder layer's lowering, so a 16-layer Llama holds ~16x copies of scratch that is only live during a single op — a large slice of the device compute/scratch footprint. Solution: add an acquire/release reuse pool for SINGLE-OP-LIFETIME scratch. `acquire_scratch()` returns a free slot (best-fit, size in `[n, 2n]`) or creates one; the caller releases it at op-lowering scope exit via the RAII `ScopedScratch` guard. Across the layers the SDPA/FD scratch now recycles a small constant of buffers instead of Nx. Whole-graph-lifetime scratch stays on `create_scratch_buffer` unchanged. Correctness: reuse is bit-identical because WebGPU/Dawn auto-inserts RAW hazard barriers between dispatches that share a storage buffer regardless of pass structure — the same guarantee the shipped `mem_obj_id` tensor aliasing already relies on. A still-`in_use` slot is never handed to a co-live requester (co-live safety). `WEBGPU_NO_SCRATCH_POOL` falls back to the per-call path for A/B. ghstack-source-id: 401515183 @exported-using-ghexport Differential Revision: [D110948389](https://our.internmc.facebook.com/intern/diff/D110948389/)
Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)
…eel GEMM Pull Request resolved: #20799 Locks the `group_size % BK` gate that selects the packed-word-dequant (`pwdq`) f16 steel GEMM. `pwdq` is bit-exact to the `half` kernel, so the existing `steel_f16`/`steel_f16_edge` configs (gs=32) already exercise it; these add the two group sizes the gate keys on but those omit. Key changes: - `test_quantized_linear.py` / `test_webgpu_native.cpp` — add `pwdq_gs64` (gs=64, `% BK == 0` → stays on `pwdq`) and `pwdq_gs8` (gs=8 `< BK=16` → falls back to the per-nibble `half` kernel, whose per-K scale read is correct there). Both goldened at the f16 tol (2.3e-4) in the `WGPU_BACKEND_STEEL_F16` build against the fp64 dequant-matmul truth. Co-authored-with: Claude Code. ghstack-source-id: 401564881 @exported-using-ghexport Differential Revision: [D111163551](https://our.internmc.facebook.com/intern/diff/D111163551/)
…GEMM Pull Request resolved: #20800 **+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).** **Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple. **Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias. Before (`pwdq`): f16 multiply, f32 accumulate. After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue. **Implementation:** - New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file. - Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag. - When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed). **Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments. Co-authored-with: Claude Code. ghstack-source-id: 401564900 @exported-using-ghexport Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Pull Request resolved: #20801 Adds golden coverage for the f16-accumulate (`pwdqf16acc`) steel GEMM, which runs under `WGPU_BACKEND_STEEL_F16ACC` when `group_size % BK == 0`. Key changes: - `test_quantized_linear.py` / `test_webgpu_native.cpp` — add `pwdqf16acc` (96x2048x256, K=2048) and `pwdqf16acc_down` (128x8192x2048, deep-K worst case) under `#ifdef WGPU_BACKEND_STEEL_F16ACC`, goldened against the fp64 dequant-matmul truth. f16 accumulation error grows with K, so the tolerances are wider than the f16-multiply `steel_f16` (2.3e-4) and the deep-K shape gets the loosest gate; perplexity (the kernel diff) is the primary quality bar and this catches gross bit/index bugs. Co-authored-with: Claude Code. ghstack-source-id: 401515200 @exported-using-ghexport Differential Revision: [D111163651](https://our.internmc.facebook.com/intern/diff/D111163651/)
meta-cla
Bot
added
the
CLA Signed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was created by the merge bot to help merge the original PR into the main branch.
ghstack PR number: #20773 by @JCNTH
^ Please use this as the source of truth for the PR details, comments, and reviews
ghstack PR base: https://github.com/pytorch/executorch/tree/gh/JCNTH/11/base
ghstack PR head: https://github.com/pytorch/executorch/tree/gh/JCNTH/11/head
Merge bot PR base: https://github.com/pytorch/executorch/tree/gh/JCNTH/10/orig
Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/JCNTH/11/orig
@diff-train-skip-merge