Add SSL_CTX_add_client_custom_ext (OpenSSL-compat client custom extensions) by julek-wolfssl · Pull Request #10625 · wolfSSL/wolfssl

julek-wolfssl · 2026-06-05T18:04:47Z

Implements the OpenSSL-compatible legacy client custom extension API
(SSL_CTX_add_client_custom_ext) for TLS 1.2 and below.

Application-defined extensions carry arbitrary IANA types, which cannot live
in the TLSX list (it keys every extension on a fixed 72-bit semaphore index
that an arbitrary type would overrun), so they are kept in a separate list on
the WOLFSSL_CTX and processed alongside the unknown-extension handling,
mirroring OpenSSL's custext.

Commits

Add SSL_CTX_add_client_custom_ext — registers a {add,free,parse}
method set with validation. ClientHello runs each add_cb and serializes
the wire bytes; ServerHello dispatches the unknown-type case to parse_cb.
Gated to non-TLS-1.3 so TLS 1.3 keeps RFC 8446 unsupported_extension
behavior. Lists/buffers freed in SSL_CtxResourceFree/SSL_ResourceFree;
openssl/ssl.h maps the API and custom_ext_*_cb typedefs.
Reject unsolicited/duplicate exts and ignore on resumption — tracks the
custom types emitted in the ClientHello and rejects a server echo of an
unsent type (unsupported_extension), suppresses parsing on a resumed
handshake (SSL_EXT_IGNORE_ON_RESUMPTION), and scans for duplicate custom
types (DUPLICATE_TLS_EXT_E), matching extensions_cust.c/extensions.c.
Confirmed-resumption gate and flexible client methods — gates the
resumption ignore on the server-confirmed signal (resumption attempted AND
the server echoed our session ID), and always offers the extension in the
ClientHello regardless of the method's max version, so
wolfSSLv23_client_method()/wolfTLS_client_method() offer it when
negotiating TLS 1.2. The negotiated-version restriction remains enforced on
the parse side.

Adds unit tests covering registration validation, TLS 1.2/1.3 and
flexible-method handshakes (add/free callbacks balance), ServerHello parse
dispatch, and unsolicited/duplicate/resumption handling. Builds with and
without OPENSSL_EXTRA; full API suite passes.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Implements OpenSSL-compatible legacy client custom extensions (SSL_CTX_add_client_custom_ext) for TLS 1.2 and below, including registration, ClientHello serialization, ServerHello parse dispatch, and related validation behaviors.

Changes:

Added public API + OpenSSL-compat typedef/macro mappings for client custom extensions.
Introduced internal storage and processing for app-defined custom extensions alongside unknown-extension handling.
Added unit tests for registration validation, handshake behavior across TLS 1.2/1.3, parse dispatch, and unsolicited/duplicate/resumption handling.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
wolfssl/ssl.h	Adds public callback typedefs and `wolfSSL_CTX_add_client_custom_ext` API declaration.
wolfssl/openssl/ssl.h	Maps OpenSSL names (`custom_ext_*_cb`, `SSL_CTX_add_client_custom_ext`) onto wolfSSL API.
wolfssl/internal.h	Introduces `WOLFSSL_CustomExt` and adds per-CTX/per-SSL fields for custom ext wire bytes and sent-type tracking.
src/tls.c	Implements custom ext registration, ClientHello build/write integration, parse dispatch, and duplicate detection.
src/internal.c	Frees custom ext lists/buffers during ctx/ssl teardown.
tests/api/test_tls_ext.h	Exposes new custom-ext test declarations.
tests/api/test_tls_ext.c	Adds comprehensive unit tests for custom-ext behaviors.
tests/api.c	Registers new test cases in the test runner.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Address the second round of review on wolfSSL#10625: - TLSX_CustomExt_BuildRequest: if an add_cb returns 1 with outlen > 0 but leaves *out == NULL, reject with BAD_FUNC_ARG (running free_cb for cleanup) instead of dereferencing NULL in the XMEMCPY. - TLSX_GetRequestSize: the extensions accumulator is a word16, so a large custom extension could wrap "length += customSz" before the block-length check and under-report the buffer size, risking an out-of-bounds write in TLSX_WriteRequest. Reject an oversized total using a wider type before the addition. Adds test_wolfSSL_custom_ext_add_null covering the NULL-pointer path (clean failure plus free_cb invocation).

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Address the third round of review on wolfSSL#10625: - TLSX_CustomExt_BuildRequest: validate ssl and pSz, returning BAD_FUNC_ARG for NULL input instead of dereferencing it (it is a WOLFSSL_TEST_VIS API). - TLSX_CustomExt_Parse: TLS session IDs are variable length (0..ID_LEN), so compare the actual lengths when detecting a confirmed resumption rather than hard-requiring both to equal ID_LEN, which missed a server echo of a shorter session ID. - TLSX_Parse custom-ext duplicate scan: use a word32 cursor so the "scan += 4 + extLen" step cannot wrap on untrusted input and spin or read out of bounds. Extends test_wolfSSL_custom_ext_add_null with the NULL-argument cases.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

… build Address the fourth round of review on wolfSSL#10625: - TLSX_CustomExt_IsKnown(): also reject TLSXT_ECH_OUTER_EXTENSIONS (0xfd00), which wolfSSL already parses internally, so an application cannot register a custom handler that would shadow it. - test_tls_ext.c: the custom-ext test helpers were gated on !WOLFSSL_NO_TLS12, but test_wolfSSL_custom_ext_tls13_handshake() compiles whenever TLS 1.3 and the memio deps are present. In a TLS-1.3-only build that left test_custom_ext_reset()/ctx_ready and the shared callbacks undefined. Widen the helper guard to also cover the TLS-1.3 memio case, and keep test_custom_ext_add_null_cb (used only by a TLS-1.2 test) under !WOLFSSL_NO_TLS12 to avoid an unused-function warning.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

github-actions · 2026-06-08T18:28:35Z

retest this please

github-actions · 2026-06-08T22:40:11Z

MemBrowse Memory Report

gcc-arm-cortex-m0plus

FLASH: .text +20 B (+0.0%, 63,495 B / 262,144 B, total: 24% used)

gcc-arm-cortex-m3

FLASH: .text +16 B (+0.0%, 121,269 B / 262,144 B, total: 46% used)

gcc-arm-cortex-m4

FLASH: .text +64 B (+0.0%, 198,894 B / 262,144 B, total: 76% used)

gcc-arm-cortex-m4-openssl-compat

FLASH: .rodata +24 B, .text +1,152 B (+0.2%, 767,940 B / 1,048,576 B, total: 73% used)

gcc-arm-cortex-m7-pq

FLASH: .text +64 B (+0.0%, 278,168 B / 1,048,576 B, total: 27% used)

linuxkm-standard

Data: __patchable_function_entries +40 B (+0.1%, 46,000 B)
No memory changes detected for:
gcc-arm-cortex-m4-baremetal
gcc-arm-cortex-m4-crypto-only
gcc-arm-cortex-m4-dtls13
gcc-arm-cortex-m4-min-ecc
gcc-arm-cortex-m4-pkcs7
gcc-arm-cortex-m4-pq
gcc-arm-cortex-m4-rsa-only
gcc-arm-cortex-m4-sp-math
gcc-arm-cortex-m4-tls12
gcc-arm-cortex-m4-tls13
gcc-arm-cortex-m7
gcc-arm-cortex-m7-tls13
linuxkm-pie
stm32-sim-stm32h753

julek-wolfssl · 2026-06-09T22:33:38Z

retest this please no history

…sions) Implements the OpenSSL-compatible legacy client custom extension API for TLS 1.2 and below, mirroring SSL_CTX_add_client_custom_ext(). Application- defined extensions carry arbitrary IANA types, which cannot live in the TLSX list (it keys every extension on a fixed semaphore index that an arbitrary type would overrun), so they are kept in a separate list on the WOLFSSL_CTX and processed alongside the unknown-extension handling, like OpenSSL's custext. Gated on HAVE_TLS_EXTENSIONS && OPENSSL_EXTRA. Registration (wolfSSL_CTX_add_client_custom_ext): - Validates ext_type <= 0xffff, no free_cb without add_cb, not a type wolfSSL handles internally (incl. ech_outer_extensions), no duplicates. ClientHello (send): - TLSX_GetRequestSize runs each add_cb, serializes the wire bytes into a per-connection cache and calls free_cb; TLSX_WriteRequest copies them out. Honors add_cb returns 1/0/-1 (+alert). free_cb runs only after add_cb returns 1, and through a single cleanup path. The extension is always offered regardless of the client's max version (matching OpenSSL, whose is_tls13 check is false while building the ClientHello), so it works with flexible client methods that negotiate down to TLS 1.2. - Guards against word16 wire-field and extensions-block overflow. ServerHello (parse), matching ssl/statem/extensions_cust.c: - Unsolicited: a custom extension whose type was not emitted in our ClientHello (tracked in ssl->customExtSent) is rejected with unsupported_extension (RFC 5246 7.4.1.4 / SSL_EXT_FLAG_SENT). - Resumption: like SSL_EXT_IGNORE_ON_RESUMPTION, a server echo is ignored only on server-confirmed resumption -- resumption attempted AND the server echoed our session ID (RFC 5246, RFC 5077). A full-handshake fallback still parses/validates and rejects an unsolicited extension. - Duplicates: the semaphore-based detection cannot cover arbitrary types, so scan the already-parsed portion for an earlier extension of the same registered type and abort with DUPLICATE_TLS_EXT_E. - TLS 1.3 is left to RFC 8446 unsupported_extension handling; the legacy API is TLS 1.2-and-below only, matching OpenSSL's SSL_EXT_TLS1_2_AND_BELOW_ONLY. Plumbing: - Public API and custom_ext_*_cb typedefs in ssl.h, OpenSSL-compat mapping in openssl/ssl.h, both gated on OPENSSL_EXTRA && HAVE_TLS_EXTENSIONS. - TLSX_CustomExt_BuildRequest exported WOLFSSL_TEST_VIS for unit tests. - Lists/buffers freed in SSL_CtxResourceFree and SSL_ResourceFree. Adds unit tests covering registration validation, NULL-argument paths, TLS 1.2 / flexible-method / TLS 1.3 handshakes, ServerHello parse dispatch, and the unsolicited, duplicate, and resumption (ignore and fallback) cases. Builds with and without OPENSSL_EXTRA; full API suite passes.

…ce-text The check-source-text analyzer extracts macro-like tokens from 'defined (token)' sequences anywhere in tracked files, including prose in comments. The phrase 'application-defined (custom) extensions' made it report: unrecognized macros used: custom. Reorder the wording to 'custom (application-defined)' to keep the analyzer happy.

PRB nodes intermittently fail google.test with 'tcp connect failed: Connection timed out' while www.google.com still answers ping and www.wolfssl.com:443 (external.test) connects fine: Google drops or throttles TCP connections from busy CI egress IPs, so the existing ping reachability guard does not catch it. Failing to even open the TCP connection exercises no wolfSSL code, so treat it like the unreachable-server case and skip (77) instead of failing. TLS-level failures still fail the test.

Copilot AI review requested due to automatic review settings June 5, 2026 18:04

julek-wolfssl self-assigned this Jun 5, 2026

Copilot started reviewing on behalf of julek-wolfssl June 5, 2026 18:05 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread wolfssl/ssl.h Outdated

Comment thread src/tls.c

Comment thread src/tls.c Outdated

Comment thread tests/api/test_tls_ext.c Outdated

Comment thread tests/api/test_tls_ext.h Outdated

julek-wolfssl requested a review from Copilot June 8, 2026 13:06

Copilot started reviewing on behalf of julek-wolfssl June 8, 2026 13:06 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread src/tls.c

Comment thread src/tls.c

julek-wolfssl force-pushed the client-custom-ext branch from 2fc5757 to 1de6045 Compare June 8, 2026 14:37

julek-wolfssl requested a review from Copilot June 8, 2026 14:38

Copilot started reviewing on behalf of julek-wolfssl June 8, 2026 14:38 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread src/tls.c

Comment thread src/tls.c

Comment thread src/tls.c

julek-wolfssl requested a review from Copilot June 8, 2026 15:37

Copilot started reviewing on behalf of julek-wolfssl June 8, 2026 15:38 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread src/tls.c

Comment thread tests/api/test_tls_ext.c Outdated

julek-wolfssl requested a review from Copilot June 8, 2026 16:19

Copilot started reviewing on behalf of julek-wolfssl June 8, 2026 16:20 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread src/tls.c

julek-wolfssl marked this pull request as ready for review June 8, 2026 18:26

julek-wolfssl force-pushed the client-custom-ext branch from aa5a03b to 9eed54e Compare June 9, 2026 13:55

julek-wolfssl force-pushed the client-custom-ext branch 2 times, most recently from 9f6267e to 10c1615 Compare June 10, 2026 12:46

julek-wolfssl force-pushed the client-custom-ext branch from 10c1615 to 12d466b Compare June 10, 2026 20:15

Conversation

julek-wolfssl commented Jun 5, 2026

Commits

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MemBrowse Memory Report

gcc-arm-cortex-m0plus

gcc-arm-cortex-m3

gcc-arm-cortex-m4

gcc-arm-cortex-m4-openssl-compat

gcc-arm-cortex-m7-pq

linuxkm-standard

Uh oh!

julek-wolfssl commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 8, 2026 •

edited

Loading