ML-DSA AVX512: Add new assembly

[SparkiDev]

Description

Add implementation of assembly code for AVX512F/BW and AVX512F/BW/VBMI.
Improvements to AVX2 assembly.
Check AVX512BW CPU id flag for base AVX512 assembly for ML-KEM.

Testing

Regression tested ML-DSA.

[github-actions]

MemBrowse Memory Report

linuxkm-pie

• Data: __patchable_function_entries +32 B (+0.1%, 26,440 B)

linuxkm-standard

• Data: __patchable_function_entries +32 B (+0.1%, 49,224 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11032

Scan targets checked: wolfcrypt-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src
Findings: 1

Medium (1)

mlkem_rej_uniform*_ins dispatch on AVX512F alone, not AVX512F+BW

File: wolfcrypt/src/wc_mlkem_poly.c:2327
Function: mlkem_rej_uniform_n_ins
Category: Logic errors

mlkem_rej_uniform_n_ins/mlkem_rej_uniform_ins gate the base AVX512 path with IS_INTEL_AVX512(cpuid_flags) instead of the USE_INTEL_AVX512 (F+BW) macro this PR introduced for exactly this purpose. The fallback mlkem_rej_uniform_n_avx512/mlkem_rej_uniform_avx512 routines execute vpshufb zmm..., an AVX512BW-only instruction, causing SIGILL on AVX512F-without-BW hardware. Every sibling dispatcher in this same file (mlkem_cbd_eta2_ins, mlkem_compress_10, etc.) correctly uses USE_INTEL_AVX512.

Recommendation: Change both if (IS_INTEL_AVX512(cpuid_flags)) guards to if (USE_INTEL_AVX512(cpuid_flags)) to match the rest of the file.

Referenced code: wolfcrypt/src/wc_mlkem_poly.c:2327-2331 (5 lines)

---

This review was generated automatically by Fenrir. Findings are non-blocking.

[dgarske]

Please resolve merge conflicts

[dgarske]

Results — PR #11032 (ML-DSA AVX512: Add new assembly)

Correctness — make check FAIL 0:

┌──────────────────────────────────┬────────────────────────┬─────────────────────────┐
│             Machine              │     Path exercised     │         Result          │
├──────────────────────────────────┼────────────────────────┼─────────────────────────┤
│ tigerlake-come (AVX512F/BW/VBMI) │ ML-DSA AVX512 asm runs │ PASS (after + before)   │
├──────────────────────────────────┼────────────────────────┼─────────────────────────┤
│ i9w680 (AVX2-only)               │ CPUID fallback         │ PASS (FAIL 0 / ERROR 0) │
└──────────────────────────────────┴────────────────────────┴─────────────────────────┘

Asm confirmed linked (tigerlake after): L_mldsa_avx512_perm20/21/31, permn0/1, zetas_1p, zetas_inv_1p, half_1p (796 avx512 symbols). Config: --enable-intelasm --enable-dilithium --enable-mlkem.

Speedup (tigerlake, core-pinned, 3-run avg, before -> after ops/s):

┌───────────┬───────────────────────┬───────────────────────┬───────────────────────┐
│   Level   │        key gen        │         sign          │        verify         │
├───────────┼───────────────────────┼───────────────────────┼───────────────────────┤
│ ML-DSA-44 │ 54.2k -> 81.8k (+51%) │ 17.8k -> 22.5k (+26%) │ 49.2k -> 67.1k (+36%) │
├───────────┼───────────────────────┼───────────────────────┼───────────────────────┤
│ ML-DSA-65 │ 32.5k -> 49.2k (+52%) │ 11.2k -> 16.4k (+46%) │ 31.1k -> 44.8k (+44%) │
├───────────┼───────────────────────┼───────────────────────┼───────────────────────┤
│ ML-DSA-87 │ 21.0k -> 33.5k (+60%) │ 9.6k -> 14.6k (+52%)  │ 20.0k -> 29.9k (+50%) │
└───────────┴───────────────────────┴───────────────────────┴───────────────────────┘

Every ML-DSA operation at every security level improves — +26% to +60%. No regressions.