Erlang/OTP ssl-10.2 vulnerability explained

Posted 2021-02-14 11:55:43.909512

Erlang/OTP 23.2.2 was released about a month ago, fixing a severe certificate verification vulnerability. If you are still using OTP 23.2 or 23.2.1, please upgrade now.

In this post I will demonstrate how the vulnerability can be exploited, and I will examine the root cause. But let’s start with a quick demo…

Quick demo

Since you should no longer have a vulnerable Erlang/OTP version installed, we’ll be spinning up a Docker container of Erlang/OTP 23.2.1 for our experiments. We’re going to need a CA trust store, so we’ll install the ca-certificates package before starting an Erlang shell:

$ docker run -it --rm hexpm/erlang:23.2.1-ubuntu-focal-20201008
root@40ccfb8c5f05:/# apt-get update && apt-get install ca-certificates -y
[...snip...]
done.
root@40ccfb8c5f05:/# erl
Erlang/OTP 23 [erts-11.1.5] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Eshell V11.1.5  (abort with ^G)
1>

Now let’s connect to a test server with a fake certificate, specifying the “DigiCert Global Root CA” as the trusted root:

1> ssl:start().
ok
2> ssl:connect("demo.voltone.net", 443, [{verify, verify_peer}, {cacertfile, "/usr/share/ca-certificates/mozilla/DigiCert_Global_Root_CA.crt"}]).
{ok,{sslsocket,{gen_tcp,#Port<0.7>,tls_connection,undefined},
               [<0.114.0>,<0.113.0>]}}
3> 

That seems to work, but if you try connecting to https://demo.voltone.net/ with a browser you’ll notice that the server’s certificate was not signed by DigiCert at all. Unaffected OTP versions (23.1 or earlier, 23.2.2 or later) also abort the TLS handshake:

2> ssl:connect("demo.voltone.net", 443, [{verify, verify_peer}, {cacertfile, "/usr/share/ca-certificates/mozilla/DigiCert_Global_Root_CA.crt"}]).
{error,{tls_alert,{bad_certificate,"TLS client: In state wait_cert_cr at ssl_handshake.erl:1874 generated CLIENT ALERT: Fatal - Bad Certificate\n"}}}

It is time to dig a little deeper…

Faking a certificate chain

Let’s have a closer look at the server’s certificate chain, by running openssl s_client -connect demo.voltone.net:443:

Certificate chain
 0 s:CN = demo.voltone.net
   i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA

The server claims to have a certificate for the demo.voltone.net domain, issued by a DigiCert intermediate CA, which itself was issued by the “DigiCert Global Root CA”. We did specify that CA certificate as our trust anchor, but something must be wrong.

To examine the certificates we can save them to a file using the following OpenSSL command: echo | openssl s_client -connect demo.voltone.net:443 -showcerts >chain.pem. We can then read the certificate chain and the root CA from Erlang:

1> {ok, ChainPEM} = file:read_file("chain.pem").
2> {ok, RootPEM} = file:read_file("/usr/share/ca-certificates/mozilla/DigiCert_Global_Root_CA.crt").
3> [Server, Intermediate, Root] = [DER || {'Certificate', DER, not_encrypted} <-
  public_key:pem_decode(<<ChainPEM/binary, RootPEM/binary>>)].

Let’s see if the chain is valid (using some ugly code to extract the public key from a certificate, because record expansion does not necessarily increase readability here):

4> public_key:pkix_is_issuer(Server, Intermediate).
true
5> public_key:pkix_is_issuer(Intermediate, Root).
true
6> IntermediatePK = element(3, element(8, element(2, public_key:pkix_decode_cert(Intermediate, otp)))).
7> public_key:pkix_verify(Server, IntermediatePK).
true
8> RootPK = element(3, element(8, element(2, public_key:pkix_decode_cert(Root, otp)))).
9> public_key:pkix_verify(Intermediate, RootPK).
false

Ah! So public_key:pkix_is_issuer/2 seems to confirm that the intermediate CA certificate was issued by the root CA, but public_key:pkix_verify/2 shows that the signature does not match. It turns out the entire chain is fake, the Subject and Issuer values have been chosen to resemble real DigiCert certificates, but the DigiCert root CA did not actually sign the intermediate. So why did OTP 23.2.1 trust the certificate?

Chain validation in ssl

Essentially certificate validation is performed by ssl in two stages: first a candidate chain is built, and then that chain is verified. As long as the second stage performs all the necessary checks, the first stage can take shortcuts. Since the first stage may need to iterate over many possible early candidates, taking shortcuts helps improve performance, filtering down the list of candidates as early as possible and performing expensive crypto operations only on those candidates that passed the initial screening.

One cheap way to check whether one certificate could be the issuer of another is to check whether the respective Subject and Issuer fields match, which is what public_key:pkix_is_issuer/2 does. But the X509 specification defines a more robust mechanism, using the Authority Key Identifier (AKI) extension.

OTP’s ssl application only considers the authorityCertIssuer and authorityCertSerialNumber of the AKI extension value, and (unfortunately) ignores the keyIdentifier field. While all CAs these days include the AKI extension in certificates they issue, many CAs only specify the keyIdentifier, and in such cases OTP falls back to using public_key:pkix_is_issuer/2, followed by the (expensive) public_key:pkix_verify/2 check; all still in stage one.

The “DigiCert TLS RSA SHA256 2020 CA1” fake certificate sent by the server does include an AKI extension with issuer and serial number, as we can verify with openssl x509 -text -noout:

  X509v3 Authority Key Identifier: 
    keyid:E2:8B:B0:5B:6E:5B:F4:64:E9:B5:6A:30:C2:16:3E:64:AC:9A:85:AC
    DirName:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
    serial:08:3B:E0:56:90:42:46:B1:A1:75:6A:C9:59:91:C7:4A

If the AKI had only specified a keyid, or if the extension had been absent, OTP would have had to fallback to pkix_is_issuer/2 and pkix_verify/2 to look for the issuing root CA, and it would not have found a match. But because the DirName and serial fields are present and, crucially, match the values of the legitimate “DigiCert Global Root CA” in the trust store, a certificate chain candidate is found and is passed to the second stage for verification.

An unfortunate case of copy & paste

Normally the second stage would verify the entire chain, according to the procedures defined elsewhere in the same RFC, and the signature mismatch on the intermediate CA would be detected. Unfortunately, a bug introduced in ssl version 10.2 here effectively disabled these checks for the top-level intermediate CA.

I first noticed this when I accidentally misconfigured a local test case with too low a value for the pathLenConstraint parameter in the Basic Constraints extension. The test was passing with OTP 23.2, but failing with earlier versions. Some further testing showed that OTP 23.2 also did not check expiry or key usage of that first CA certificate. But attempts to fake the signature on that certificate initially failed.

It turned out that it was actually the first stage of certificate chain building that detected the signature mismatch. After reading through the changes introduced in OTP 23.2 and some other ssl sources I realized I could produce an intermediate that did pass the first stage, by preparing the AKI extension as shown above. With this, ssl 10.2 would accept completely fake certificate chains, requiring only knowledge of the Subject and serial number of a root CA certificate trusted by the victim.

The bug appears to be a result of an unfortunate copy & paste from line 473 to line 485. As a result, the chain that’s being passed to the second stage validation is cut short, and the first intermediate CA (the last certificate sent by the server) is treated as the trust anchor. Instead, the newly found IssuerCert should be treated as the trust anchor, and the entire chain in the Path variable should be passed to the second stage for validation:

  case ssl_manager:lookup_trusted_cert(CertDbHandle, CertDbRef, SerialNr, IssuerId) of
      {ok, {NewIssuerCert, _}} ->  
          case public_key:pkix_is_self_signed(NewIssuerCert) of
-             true -> %% IssuerCert = ROOT
-                maybe_shorten_path(Path, PartialChainHandler, {IssuerCert, Rest});
+             true -> %% NewIssuerCert = ROOT
+                maybe_shorten_path([NewIssuerCert | Path], PartialChainHandler,
+                                       {NewIssuerCert, Path});
              false ->
                  maybe_shorten_path([NewIssuerCert | Path],