Add ARM64 crypto as NEON C intrinsics for MSVC (WOLFSSL_ARMASM_INTRINSICS) - #11035
Add ARM64 crypto as NEON C intrinsics for MSVC (WOLFSSL_ARMASM_INTRINSICS)#11035coneco-cy wants to merge 2 commits into
Conversation
…SICS)
wolfSSL's ARM64 crypto kernels ship in two forms: GNU/GAS assembly with GCC
inline asm (armv8-*-asm.S / -asm_c.c), and Microsoft-syntax assembly assembled
by armasm64.exe (armv8-*-asm.asm). The armasm64 route covers MSVC ARM64 and is
wired up in wolfssl.vcxproj via WolfSSLAarch64Asm.
That route needs a separate assembler invocation, which in practice is only
wired up through MSBuild CustomBuild rules. Projects consuming wolfSSL through
CMake - or any build without an ARM64 assembler step - cannot use it as-is, and
fall back to portable C for AES, AES-GCM (including table-based GHASH),
SHA-256, Poly1305 and ChaCha20.
This adds a third form: the same kernels as portable NEON C intrinsics, which
cl.exe compiles as ordinary C. No assembler step, no generated .asm, and no
per-source target-feature flag (MSVC enables NEON crypto unconditionally on
ARM64, unlike clang which needs -march=armv8-a+crypto).
The four new files supply the same *_AARCH64 symbols as the assembly, so they
are an alternative to it rather than an addition. They are opt-in behind
WOLFSSL_ARMASM_INTRINSICS and each is self-guarding on
_MSC_VER && !__clang__ && _M_ARM64, so every existing configuration and
toolchain is unaffected.
armv8-aes-intrinsics-msvc.c 19 AES/AES-GCM/AES-CBC *_AARCH64 kernels
plus 7 base-layer fallback symbols
armv8-sha-intrinsics-msvc.c Transform_Sha256_Len_crypto
armv8-poly1305-intrinsics-msvc.c 4 Poly1305 kernels
armv8-chacha-intrinsics-msvc.c 4 ChaCha20 kernels, 4-block-parallel
No existing crypto source is modified: settings.h already bridges
_M_ARM64 to __aarch64__ under WOLFSSL_ARMASM, and cpuid.c already has a
Windows ARM64 IsProcessorFeaturePresent() path, so the gates and the runtime
feature probe work as-is.
Each kernel was verified against three independent references: the original
inline asm extracted verbatim and built with clang-cl, independent pure-C
oracles, and published KATs (FIPS-197, FIPS 180-4, SP800-38A CBC/CTR, NIST
GCM, RFC 8439). Every test's failure path was proven first with an injected
negative control.
Not yet ported: AES_XTS_* (WOLFSSL_AES_XTS) and the _EOR3 variants
(WOLFSSL_ARMASM_CRYPTO_SHA3). A verified SHA-512 kernel is included but left
inert pending a wired-up feature gate.
|
Can one of the admins verify this patch? |
|
Hi @coneco-cy , thank you for submitting this work. I've asked @SparkiDev to review it and see if this is something we'd consider accepting. Can you tell us more about your project and interest in wolfSSL? If we are to accept this we will need to get a signed contributor agreement in place. Thanks, David Garske, wolfSSL |
The first version of these kernels was correct but measurably slower than the armasm64 path. Profiling the generated code on a Snapdragon X2 Elite (Oryon) found two distinct causes, each isolated with a controlled experiment: 1. The rounds were driven by a runtime `nr`, so MSVC emitted them as a real loop - 3 scalar bookkeeping ops (sub/add/cbnz) per 2 AES instructions. At a fixed 1 block in flight, unrolling alone measured 2732 -> 5913 MB/s (2.16x). Fixed by dispatching on nr to three compile-time-constant bodies; AES has only three key sizes, so three instantiations cover every case. 2. ECB/CTR processed one block at a time, which is latency-bound: aese->aesmc is a serial chain with nothing to interleave. A pipeline-depth sweep (same instruction mix per block, only the number of independent chains varying) measured 2503 / 3540 / 4310 / 8024 MB/s at 1 / 2 / 4 / 8 blocks. Upstream's assembly avoids this with its _start_2/4/8 tiers; this now does 8 wide. A third, non-obvious problem showed up while fixing (2): with the eight states passed as `uint8x16_t s[8]`, MSVC keeps them in MEMORY and still emits a single-block chain - the disassembly showed 42 load/stores for 63 AES ops and the wide path was no faster. The lanes must be named locals for the compiler to keep them in v-registers, hence the AES_X8_* macros. Similarly the single-block helpers need __forceinline: the three unrolled bodies push them past MSVC's inline heuristic, which otherwise emits a `bl` per block. Measured after the change, on an idle machine with a 20s cooldown between every run (sustained back-to-back runs thermally downclock this part from 4454 to 1382 MHz, which silently distorts any comparison), 4 interleaved pairs, 256 MiB per case, checksums verified identical between the two builds: case intrinsics armasm64 asm/intrin AES-256-ECB 5571 [5376-5658] 5638 [5480-5750] 1.012 (overlap) AES-256-CTR 6875 [6294-7136] 7057 [6610-7269] 1.026 (overlap) AES-256-CBC 1179 [1149-1210] 1090 [1084-1096] 0.924 GHASH block 1905 [1849-1943] 1964 [1955-1982] 1.031 ECB was 1.97x slower before this change; it is now within run-to-run variance. No case is now at a material disadvantage to the hand-written assembly. Correctness was re-verified against the armasm64 kernels linked into the same binary (all three key sizes, block counts 1..20 covering every remainder class around the new 8-block boundary, 63/94/125-block sizes, CTR partial tails exercising tmp/*left, ECB decrypt) plus the FIPS-197 C.1 and SP800-38A F.5.1 vectors, and GHASH separately over 1..40 folded blocks. Each check was first proven able to fail by injecting a negative control (dropping round 12 from the unrolled body: 79 failures; reversing the 8-block store order: 87 failures; GHASH reduction constant 0x87->0x86: red). Also moves two declarations in AES_GCM_ghash_block_AARCH64 to the top of the block: clang-cl's -Wdeclaration-after-statement flagged them as pre-C99 incompatible.
|
retest this please |
Hi @dgarske , Happy to give the background, since it explains why this is shaped the way it is. I work at Qualcomm, on Windows-on-ARM64 (Snapdragon) enablement. One of our ISVs deploys MariaDB, and we are working to get MariaDB building and running well on Windows ARM64 with the MSVC toolchain. MariaDB bundles wolfSSL as its TLS/crypto provider (extra/wolfssl), so wolfSSL's ARM64 story on MSVC becomes our story too — that is the whole of my interest here. The concrete problem we hit: So on MSVC ARM64 there is no ASM language enabled and no armasm64 step, and all of AES, AES-GCM (including table-based GHASH), SHA-256, Poly1305 and ChaCha20 fall back to portable C. That was visible in profiling as crypto hot spots on an otherwise capable machine. |
Description
wolfSSL's ARM64 crypto kernels ship in two forms today:
armv8-*-asm.S/armv8-*-asm_c.c— GNU/GAS assembly and GCC inline asmarmv8-*-asm.asm— Microsoft syntax, assembled byarmasm64.exeThe armasm64 form already covers MSVC on Windows ARM64 and is wired up in
wolfssl.vcxproj(WolfSSLAarch64Asm=true). This PR does not replace it.What it adds is a third form of the same kernels — portable NEON C
intrinsics — because the armasm64 route requires a separate assembler
invocation, and in practice that is only wired up through MSBuild
CustomBuildrules. Projects that consume wolfSSL through CMake, or anybuild system without an ARM64 assembler step, cannot use it as-is and silently
fall back to portable C for AES, AES-GCM (including the table-based GHASH),
SHA-256, Poly1305 and ChaCha20.
Intrinsics are ordinary C, so
cl.execompiles them directly: no assemblerstep, no generated
.asm, and no per-source target-feature flag (MSVC enablesNEON crypto unconditionally on ARM64, unlike clang which needs
-march=armv8-a+crypto).This came out of profiling MariaDB — which bundles wolfSSL and builds it with
CMake — on a native Windows ARM64 machine, where GHASH and AES showed up as the
top crypto hot spots purely because the hardware path was unreachable from that
build system.
What is added
Four new sibling files under
wolfcrypt/src/port/arm/, plus 44 lines of buildwiring. No existing crypto source is modified.
armv8-aes-intrinsics-msvc.c*_AARCH64kernels, plus the 7 base-layer symbolsaes.clinks unconditionallyarmv8-sha-intrinsics-msvc.cTransform_Sha256_Len_cryptoarmv8-poly1305-intrinsics-msvc.cpoly1305_set_key,poly1305_arm64_block_16,poly1305_arm64_blocks,poly1305_finalarmv8-chacha-intrinsics-msvc.cwc_chacha_setkey,wc_chacha_setiv,wc_chacha_use_over,wc_chacha_crypt_bytes— 4-block-parallelThese supply the same
*_AARCH64symbols as the assembly, so they are analternative to it, not an addition — hence an explicit opt-in rather than an
automatic fallback. Two independent layers keep existing builds untouched:
WOLFSSL_ARMASM_INTRINSICSoption, default off, whichFATAL_ERRORsif requested on a non-MSVC toolchain.
WOLFSSL_ARMASM && WOLFSSL_ARMASM_INTRINSICS && _MSC_VER && !__clang__ && (_M_ARM64 || _M_ARM64EC),so it compiles to nothing otherwise. (
!__clang__matters:clang-cldefines_MSC_VERbut ships clang'sarm_neon.h, wherevmull_p64's low form takes ascalar
poly64_trather than MSVC's__n64.)Why no changes to existing sources were needed
Worth noting explicitly, since an earlier version of this work did need them —
current master already solves both prerequisites:
settings.h:388maps
_M_ARM64→__aarch64__underWOLFSSL_ARMASM, so the ~104__aarch64__gates inaes.cselect correctly under MSVC.cpuid.c:473has a
_WIN32branch usingIsProcessorFeaturePresent(), soCheck_CPU_support_HwCrypto()works — no GCC-onlymrsprobe needed.Because
HAVE_CPUID_AARCH64derives from__aarch64__ && WOLFSSL_ARMASM, thatbridge also enables the Windows cpuid path automatically.
Testing
All on a native Windows ARM64 machine (Snapdragon X2 Elite / Oryon),
MSVC 14.44.35207, SDK 10.0.26100.0.
Each kernel was validated against three independent references, not one:
armv8-*-asm_c.cbodies were extracted verbatim by script (never retyped) into a reference TU
built with
clang-cl(cl.execannot compile GCC inline asm), then comparedover randomized inputs and edge cases — AES 100k random blocks × all three key
sizes; CBC over block counts 1/2/3/7/16/255; ChaCha over a ragged multi-call
sequence (5,1,10,64,3,100,63,65,200,7 bytes), which is the only shape that
exercises the
over/leftleftover-keystream contract.GHASH, FIPS 180-4 SHA-256/512, RFC 8439 Poly1305/ChaCha20).
(F.2.1) and AES-128-CTR (F.5.1, including a 20-byte partial tail), NIST GCM
case 4 (ciphertext + tag + valid-decrypt + corrupt-tag reject), FIPS 180-4
SHA-256/512, RFC 8439 §2.4.2 and §2.5.2.
Performance vs. portable C
In-process profiles, symbol swap visible in-trace:
Intrinsics vs. the armasm64 assembly
Since the point of this PR is an alternative to the assembly, I also compared the
performance difference between intrinsics and armasm64 assembly. Both
kernels were linked into the same binary from the same benchmark source, so only
the implementation differs.
The figures below use an idle machine, a 20 s
cooldown between every single run, discarded warm-up runs, and interleaved
A/B/A/B ordering. Checksums were verified identical between the two builds, so
both sides provably did the same work. 256 MiB per case, AES-256.
ECB and CTR are indistinguishable at this precision; CBC is ~8% faster in
intrinsics and GHASH ~3% slower. So the intrinsics are performance-equivalent
to the hand-written assembly — no case is at a material disadvantage.
Checklist
controls; not yet integrated into wolfSSL's own suite, see question 5