Error when using SHA384WITHECDSA certificate with common providers

[XyproChance]

Currently integrating BC-JSSE into some of our products for PQC compliance and certificates signed with SHA384withECDSA are being rejected when validating an external server's certificate chain. This was reproduced identically against www.google.com and www.cloudflare.com and confirmed to work with SunJSSE. This is the current error when attempting to connect that fails during the TLS 1.3 handshake:

org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
Caused by: java.security.cert.CertificateException: Unable to construct a valid chain
Caused by: java.security.cert.CertPathValidatorException:
  Signature algorithm 'SHA384WITHECDSA' not permitted with given parameters and issuer public key

• The TLS handshake succeeds through ClientHello/ServerHello key exchange and only fails at the ServerCertificate processing step which is the certificate chain validation.
• Reproduced on BC 1.79 and BC 1.85
• A control test using plain vanilla JDK/SunJSSE with the identical jdk.tls.namedGroups setting and no BouncyCastle connects successfully.

We have tested directly and ruled out issues with

• JCA provider priority/registration order
• Which TrustManagerFactory/provider validates the chain, tried using BC's own PKIX implemenation
• jdk.tls.disabledAlgorithms / jdk.certpath.disabledAlgorithms security properties
• Static initialization order (properties set after a BC provider was already constructed)

We're also fairly certain #1343 is not the problem as this occurs even when properties are fully empty.

If this is not a configuration issue or some misstep on our end I believe we have narrowed this down to either a key-size measurement bug inside BC's KeySizeConstraint logic or a separate algorithm issue inside the JDK itself. If anyone has run into something similar or can see I'm making a mistake your input would be greatly appreciated. At the bottom I have included the code being run attempting a connection to www.google.com and www.cloudflare.com as well as the control test showing a SunJSSE connection does in fact work.

This is using JDK11 and BC 1.85
Reproduction:

Dependencies used (all resolved from Maven Central):

• org.bouncycastle:bcprov-jdk18on:1.85 (also reproduced on 1.79)
• org.bouncycastle:bctls-jdk18on:1.85 (also reproduced on 1.79)
• org.bouncycastle:bcutil-jdk18on:1.85 (also reproduced on 1.79)
• io.netty:netty-handler:4.1.101.Final (and its transitive netty-* dependencies)
• io.projectreactor.netty:reactor-netty-http:1.0.39 (and its transitive dependencies)
• JDK 11 (11.0.18)

Build a classpath containing the above (e.g. via Maven dependency:build-classpath for a
project that depends on reactor-netty-http, plus the three BouncyCastle jars), then:

javac -cp PqcNettyTest.java
java -cp .; PqcNettyTest

PQCTest1.java

import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslContextBuilder;
import io.netty.handler.ssl.SslProvider;
import org.bouncycastle.jsse.provider.BouncyCastleJsseProvider;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import reactor.netty.http.client.HttpClient;

import javax.net.ssl.TrustManagerFactory;
import java.security.KeyStore;
import java.security.Security;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;

public class PqcNettyTest {
    public static void main(String[] args) throws Exception {
        // Decisive test: set the properties BEFORE constructing/registering ANY
        // BouncyCastle class at all, to rule out static-init-order latching the
        // old/default value into ProvAlgorithmConstraints' static fields.
        Security.setProperty("jdk.tls.disabledAlgorithms", "");
        Security.setProperty("jdk.certpath.disabledAlgorithms", "");

        System.out.println(">>> readback jdk.tls.disabledAlgorithms = [" + Security.getProperty("jdk.tls.disabledAlgorithms") + "]");
        System.out.println(">>> readback jdk.certpath.disabledAlgorithms = [" + Security.getProperty("jdk.certpath.disabledAlgorithms") + "]");

        Security.addProvider(new BouncyCastleProvider());
        Security.insertProviderAt(new BouncyCastleJsseProvider(), 1);

        System.setProperty("jdk.tls.namedGroups",
                "x25519,secp256r1,x25519mlkem768,secp256r1mlkem768,mlkem768");

        BouncyCastleJsseProvider bcJsse = new BouncyCastleJsseProvider();

        // Use the JDK's real cacerts trust store, same as production WebClientConfig would
        // be handed a real truststore instead of BC-JSSE's bare default trust manager.
        String cacertsPath = System.getProperty("java.home") + "/lib/security/cacerts";
        KeyStore cacerts = KeyStore.getInstance("JKS");
        try (java.io.FileInputStream fis = new java.io.FileInputStream(cacertsPath)) {
            cacerts.load(fis, "changeit".toCharArray());
        }
        TrustManagerFactory tmf = TrustManagerFactory.getInstance("PKIX", "SunJSSE");
        tmf.init(cacerts);
        System.out.println(">>> TrustManagerFactory provider: " + tmf.getProvider().getName());

        SslContext sslContext = SslContextBuilder.forClient()
                .sslProvider(SslProvider.JDK)
                .sslContextProvider(bcJsse)
                .protocols("TLSv1.3")
                .trustManager(tmf)
                .build();

        System.out.println("SslContext built OK. Provider in use: " + bcJsse.getName());
        System.out.println("Underlying isClient=" + sslContext.isClient());

        AtomicReference<String> negotiated = new AtomicReference<>("UNKNOWN");
        CountDownLatch latch = new CountDownLatch(1);

        HttpClient.create()
                .secure(spec -> spec.sslContext(sslContext))
                .doOnConnected(conn -> {
                    io.netty.handler.ssl.SslHandler handler =
                            conn.channel().pipeline().get(io.netty.handler.ssl.SslHandler.class);
                    if (handler != null) {
                        javax.net.ssl.SSLSession session = handler.engine().getSession();
                        System.out.println(">>> Negotiated protocol: " + session.getProtocol());
                        System.out.println(">>> Negotiated cipher suite: " + session.getCipherSuite());
                        System.out.println(">>> Session class: " + session.getClass().getName());
                    }
                })
                .get()
                .uri("https://www.google.com")
                .responseSingle((res, body) -> {
                    negotiated.set("HTTP status: " + res.status());
                    return body.asString().defaultIfEmpty("");
                })
                .doOnError(e -> {
                    negotiated.set("ERROR: " + e);
                    latch.countDown();
                })
                .doOnSuccess(s -> latch.countDown())
                .subscribe();

        latch.await(20, TimeUnit.SECONDS);
        System.out.println("Result: " + negotiated.get());
    }
}

PQCTest2.java

import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslContextBuilder;
import io.netty.handler.ssl.SslProvider;
import org.bouncycastle.jsse.provider.BouncyCastleJsseProvider;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import reactor.netty.http.client.HttpClient;

import javax.net.ssl.TrustManagerFactory;
import java.security.KeyStore;
import java.security.Security;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;

public class PqcNettyTest2 {
    public static void main(String[] args) throws Exception {
        Security.insertProviderAt(new BouncyCastleProvider(), 1);
        Security.insertProviderAt(new BouncyCastleJsseProvider(), 2);

        System.setProperty("jdk.tls.namedGroups",
                "x25519,secp256r1,x25519mlkem768,secp256r1mlkem768,mlkem768");

        BouncyCastleJsseProvider bcJsse = new BouncyCastleJsseProvider();

        // Use the JDK's real cacerts trust store, same as production WebClientConfig would
        // be handed a real truststore instead of BC-JSSE's bare default trust manager.
        String cacertsPath = System.getProperty("java.home") + "/lib/security/cacerts";
        KeyStore cacerts = KeyStore.getInstance("JKS");
        try (java.io.FileInputStream fis = new java.io.FileInputStream(cacertsPath)) {
            cacerts.load(fis, "changeit".toCharArray());
        }
        TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        tmf.init(cacerts);

        SslContext sslContext = SslContextBuilder.forClient()
                .sslProvider(SslProvider.JDK)
                .sslContextProvider(bcJsse)
                .protocols("TLSv1.3")
                .trustManager(tmf)
                .build();

        System.out.println("SslContext built OK. Provider in use: " + bcJsse.getName());
        System.out.println("Underlying isClient=" + sslContext.isClient());

        AtomicReference<String> negotiated = new AtomicReference<>("UNKNOWN");
        CountDownLatch latch = new CountDownLatch(1);

        HttpClient.create()
                .secure(spec -> spec.sslContext(sslContext))
                .doOnConnected(conn -> {
                    io.netty.handler.ssl.SslHandler handler =
                            conn.channel().pipeline().get(io.netty.handler.ssl.SslHandler.class);
                    if (handler != null) {
                        javax.net.ssl.SSLSession session = handler.engine().getSession();
                        System.out.println(">>> Negotiated protocol: " + session.getProtocol());
                        System.out.println(">>> Negotiated cipher suite: " + session.getCipherSuite());
                        System.out.println(">>> Session class: " + session.getClass().getName());
                    }
                })
                .get()
                .uri("https://www.cloudflare.com")
                .responseSingle((res, body) -> {
                    negotiated.set("HTTP status: " + res.status());
                    return body.asString().defaultIfEmpty("");
                })
                .doOnError(e -> {
                    negotiated.set("ERROR: " + e);
                    latch.countDown();
                })
                .doOnSuccess(s -> latch.countDown())
                .subscribe();

        latch.await(20, TimeUnit.SECONDS);
        System.out.println("Result: " + negotiated.get());
    }
}

PlainTest.java

import javax.net.ssl.HttpsURLConnection;
import java.net.URL;

public class VanillaTest {
    public static void main(String[] args) throws Exception {
        System.setProperty("jdk.tls.namedGroups",
                "x25519,secp256r1,x25519mlkem768,secp256r1mlkem768,mlkem768");
        URL url = new URL("https://www.google.com");
        HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
        conn.setSSLSocketFactory(javax.net.ssl.SSLContext.getDefault().getSocketFactory());
        System.out.println("Response code: " + conn.getResponseCode());
        System.out.println("Cipher suite: " + conn.getCipherSuite());
    }
}