Fix RSA-OAEP to allow zero-length plaintext per RFC 8017#10012
Open
MarkAtwood wants to merge 2 commits into
Open
Fix RSA-OAEP to allow zero-length plaintext per RFC 8017#10012MarkAtwood wants to merge 2 commits into
MarkAtwood wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates RSA-OAEP handling to permit zero-length plaintexts on both encryption and decryption, aligning behavior with RFC 8017 and other TLS/crypto libraries.
Changes:
- Relaxed
RsaPublicEncryptEx()argument validation to allowinLen == 0for OAEP padding. - Adjusted
RsaPrivateDecryptEx()constant-time error handling to permitret == 0results for OAEP padding. - Added RFC 8017 references in inline comments to document the behavioral change.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
MarkAtwood
force-pushed
the
fix/rsa-oaep-allow-empty-plaintext
branch
from
eefa39a to
a0752c3
Compare
MarkAtwood
force-pushed
the
fix/rsa-oaep-allow-empty-plaintext
branch
from
a0752c3 to
e8e7cd1
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Updates wolfCrypt RSA-OAEP handling to align with RFC 8017 by permitting zero-length plaintexts on encrypt and decrypt paths.
Changes:
- Adjusts
RsaPublicEncryptEx()argument validation to allowinLen == 0for OAEP padding. - Adjusts
RsaPrivateDecryptEx()handling ofret == 0to allow zero-length OAEP results (under#ifndef WOLFSSL_RSA_DECRYPT_TO_0_LEN) using constant-time masking. - Adds inline RFC 8017 references in both code paths.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
MarkAtwood
force-pushed
the
fix/rsa-oaep-allow-empty-plaintext
branch
from
e8e7cd1 to
5cbeab1
Compare
Contributor
Author
|
/cc @wolfSSL-Fenrir-bot please review — particularly the constant-time masking in the decrypt path |
LinuxJedi
requested changes
Apr 23, 2026
Member
LinuxJedi
left a comment
LinuxJedi
left a comment
There was a problem hiding this comment.
@MarkAtwood please remove the Claude co-authored-by in the commit message.
MarkAtwood
force-pushed
the
fix/rsa-oaep-allow-empty-plaintext
branch
from
5cbeab1 to
a8c6769
Compare
Encrypt side: RsaPublicEncryptEx() rejected inLen==0 unconditionally. RFC 8017 Section 7.1.1 permits zero-length messages (mLen=0 satisfies the message-length bound mLen <= k - 2*hLen - 2). Gate the inLen==0 rejection on pad_type != WC_RSA_OAEP_PAD. Decrypt side: RsaUnPad_OAEP() returned 0 on both padding errors and valid empty messages, making them indistinguishable. Change the error sentinel from pkcsBlockLen to pkcsBlockLen+1 so padding errors produce ret==-1 (via unsigned wraparound) while valid empty messages produce ret==0. In RsaPrivateDecryptEx(), use constant-time masking to allow ret==0 specifically for WC_RSA_OAEP_PAD, preserving the existing rejection of zero-length results for other padding types. The constant-time properties of the OAEP unpadding are preserved: the only change is the value of the error-path constant, not the control flow or number of operations. Both OpenSSL and BoringSSL accept empty OAEP plaintexts. Found via Wycheproof test vectors.
MarkAtwood
force-pushed
the
fix/rsa-oaep-allow-empty-plaintext
branch
from
a8c6769 to
85f796a
Compare
Contributor
|
Jenkins retest this please |
Contributor
Author
|
Co-Authored-By line was already removed in the current commit. The CHANGES_REQUESTED is stale — ready for re-review. |
- Tighten RFC 8017 comments to specify step 1b/3g scope - Rename zeroOk to zeroOkMask (byte, not int) to match ctMask* API - Add zero-length OAEP encrypt/decrypt round-trip test - Verify corrupted OAEP ciphertext returns negative error
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Encrypt side:
RsaPublicEncryptEx()rejectedinLen == 0unconditionally withBAD_FUNC_ARG. RFC 8017 Section 7.1.1 (RSAES-OAEP-ENCRYPT) permits zero-length messages: the message-length constraintmLen <= k - 2*hLen - 2is satisfied bymLen = 0. Gate theinLen == 0rejection onpad_type != WC_RSA_OAEP_PAD.Decrypt side:
RsaUnPad_OAEP()deliberately returned 0 on padding errors (as a chosen-ciphertext defense — see thectMaskSelWord32on the error path). This meant that padding errors and valid empty messages both producedret == 0, making them indistinguishable. Previously this was fine becauseRsaPrivateDecryptEx()rejected allret == 0results. But naively allowingret == 0for OAEP (to support empty messages) would silently accept invalid OAEP ciphertext as a successful empty-message decryption.Fix: Change the error sentinel in
RsaUnPad_OAEP()frompkcsBlockLentopkcsBlockLen + 1, so the return value on padding error becomes-1(via unsigned wraparound to(int)(pkcsBlockLen - (pkcsBlockLen + 1))) instead of0. Valid empty messages still return0. InRsaPrivateDecryptEx(), use constant-time masking (ctMaskEq(pad_type, WC_RSA_OAEP_PAD)) to allowret == 0specifically for OAEP, while preserving the existing rejection of zero-length results for other padding types.Constant-time properties preserved: The only change to
RsaUnPad_OAEPis the value of the constant passed toctMaskSelWord32on the error path (pkcsBlockLen + 1instead ofpkcsBlockLen). The control flow, number of operations, and timing characteristics are identical. The*outputpointer on the error path points one byte past the buffer end (pkcsBlock + pkcsBlockLen + 1), but this is safe becauseret < 0prevents any caller from dereferencing or copying from it (the copy path is guarded byret >= 0).Both OpenSSL and BoringSSL accept empty OAEP plaintexts. Found via Wycheproof test vectors.
Spec references
mLen <= k - 2*hLen - 2;mLen = 0is validMwhich may be the empty stringTest plan