fix: skip RSA_MIN_PAD_SZ check for PSS padding in RsaPublicEncryptEx

[MarkAtwood]

Summary

The RSA_MIN_PAD_SZ guard (inLen > sz - 11 → RSA_BUFFER_E) is a PKCS#1 v1.5 concept. PSS has its own length check inside RsaPad_PSS (emLen >= hLen + sLen + 2 per RFC 8017 §9.1.1) and the outer guard fires first, before RsaPad_PSS ever runs.

For keys in the range [hLen+2, hLen+10] bytes, the outer guard incorrectly returns RSA_BUFFER_E for combinations where PSS with saltLen=0 would be geometrically valid. Keys in this range are non-standard but valid — they can be loaded from external DER.

Fix: add a WC_RSA_PSS #ifdef that skips the RSA_BUFFER_E return when pad_type == WC_RSA_PSS_PAD, mirroring the existing WC_RSA_NO_PADDING exception for raw mode.

Test plan

• PSS encrypt with non-standard key sizes in [hLen+2, hLen+10] bytes no longer returns RSA_BUFFER_E
• PSS still correctly rejects combinations that exceed its own length constraint (PSS_SALTLEN_E)
• Existing RSA PSS tests unaffected

/cc @wolfSSL-Fenrir-bot please review

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adjusts RsaPublicEncryptEx so the RSA_MIN_PAD_SZ (PKCS#1 v1.5) input-length guard does not preempt PSS padding’s own size validation, enabling certain non-standard small key sizes to proceed to RsaPad_PSS.

Changes:

• Skips the RSA_BUFFER_E early-return when pad_type == WC_RSA_PSS_PAD (under WC_RSA_PSS).
• Keeps the existing exception path for WC_RSA_NO_PAD behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[MarkAtwood]

Restructured as a single compound condition — pushed f05d929. No more implicit nesting of preprocessor-guarded if-statements. Each exemption is now a && clause gated by its own #ifdef, so adding future exemptions is just another line.

[MarkAtwood]

Restructured per review feedback — the implicit preprocessor nesting is replaced with a check_min_pad flag pattern (91a1b28). Each exemption is a separate #ifdef block, and there's a single guarded return RSA_BUFFER_E. Ready for re-review.