DST Root CA X3 expiration update

Posted 2021-08-29 11:47:43.370644

A few things have happened since my previous post:

This post corrects a few statements I made in the original post, in particular regarding the impact on Hackney: if you are a Hackney (or HTTPoison, or Tesla-with-Hackney) user you’ll find some good news further below.

I won’t cover the background again in this post, so please refer to the original post to learn more about how the DST Root CA X3 expiration on September 30th may impact BEAM TLS clients that connect to servers with Let’s Encrypt certificates.

Patch packages

The Erlang/OTP team have improved the handling of alternate certificate chains and released the following patch packages:

If you are on OTP 23.3 or 24.0 I would strongly recommend you upgrade to a patched version before the end of September. Older versions are not impacted in the same way, so no patch is available. However, those versions may still experience issues after September: keep reading for details.

Note that the new logic makes the partial_chain hook used by many TLS clients superfluous in most cases: shorter chains are now recognized and selected by the built-in certificate chain verification logic.

Realistic testing

When trying to predict what will happen after September 30th, I previously had to simulate the situation using Let’s Encrypt staging certificates. As we are approaching that date, production certificates now have a validity that extends past it, allowing us to see what will happen on October 1st by simply rolling forward the clock.

Here’s how we might try an httpc connection to this blog from Erlang/OTP 23.3.4, with the clock set to October 1st:

$ docker run -it --rm hexpm/elixir:1.12.2-erlang-23.3.4-ubuntu-focal-20210325
root@c15dfd70e4c7:/# apt update && apt install -y faketime
[...snip...]
root@c15dfd70e4c7:/# faketime '2021-10-01 09:00' iex
Erlang/OTP 23 [erts-11.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Interactive Elixir (1.12.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Mix.install([:certifi])
[...snip...]
:ok
iex(2)> :httpc.request(:get, {'https://blog.voltone.net/', []}, [ssl: [verify: :verify_peer, cacertfile: :certifi.cacertfile(), depth: 2]], [])

09:01:08.453 [info]  TLS :client: In state :certify at ssl_handshake.erl:1878 generated CLIENT ALERT: Fatal - Certificate Expired

{:error,
 {:failed_connect,
  [
    {:to_address, {'blog.voltone.net', 443}},
    {:inet, [:inet],
     {:tls_alert,
      {:certificate_expired,
       'TLS client: In state certify at ssl_handshake.erl:1878 generated CLIENT ALERT: Fatal - Certificate Expired\n'}}}
  ]}}

Using faketime and a production server certificate makes it possible to test the behaviour of clients with their default ssl options, since it is not necessary to select a custom CA trust store. This is especially important when testing Hackney, as it is almost impossible to reconstruct the out-of-the-box ssl options with a custom CA trust store (or any other custom options, for that matter).

Impact

In this section I will describe the impact of the DST Root CA X3 expiration on various commonly used TLS clients, depending on the Erlang/OTP version used. When I refer to an ‘impacted’ version, I mean 23.3 versions prior to 23.3.4.5 and 24.0 versions prior to 24.0.4.

I won’t show the exact command invocation used to verify each scenario: you should be able to reproduce most scenarios yourself with Docker, faketime and Mix.install as shown above. Note that I’m using Elixir, to take advantage off Mix.install to pull in Hex packages without a need to initialize a project. If you have an Erlang project with the necessary dependencies already installed, you should be able test yourself from e.g. a Rebar3 shell.

The Docker images used are:

ssl / httpc

By default these standard library modules do not verify the server’s certificate at all. It is necessary to explicitly set the verify option to verify_peer and to pass in some other ssl options, as described here.

Here’s what you can expect if you use ssl or httpc with the minimum recommended certificate verification options (no partial_chain):

Pre-23.3:

As long as the DST Root CA X3 certificate is still present in the CA trust store, connections will succeed because these OTP versions treat expired CA certificates present in the trust store as trusted.

⚠️ Once the DST Root CA X3 certificate is removed from the CA trust store, presumably after installing the next CA trust store update after September 30th, connections to servers with Let’s Encrypt’s long certificate chain will fail, because the shorter chain rooted in ISRG Root X1 is not considered!

Affected 23.3 or 24:

⚠️ From September 30th such clients will fail to connect to servers with Let’s Encrypt’s long certificate chain, regardless of whether the CA trust store contains the DST Root CA X3, ISRG Root X1 or both certificates.

Fixed 23.3 or 24:

Connects successfully when the ISRG Root X1 certificate is present in the CA trust store, regardless of whether the (expired) DST Root CA X3 its still present. A partial chain hook is not necessary. Hurrah 🎉!

Recommendation:

I would recommend upgrading to Erlang/OTP 23.3.4.5 or 24.0.4 or later. If you are still on a pre-23.3 release it may not be so easy to upgrade at short notice; in that case you can buy some time by holding on to a CA trust store that includes DST Root CA X3 even after it expires, until you are ready to migrate to the latest 23.3 or 24.

Hackney / HTTPoison

Connects successfully when either DST Root CA X3 or ISRG Root X1 or both are present in the CA trust store, on all Erlang/OTP versions I tested.

In my previous post I suggested that Hackney’s partial chain hook would be ineffective on 23.3 and 24.0, but now that I am able to test without manually reconstructing Hackney’s ssl options I see that it handles the situation after September 30th correctly.

Edit: Please note that the above statements only apply to Hackney’s out-of-the-box default ssl options. When overriding the ssl options Hackney behaves just like ssl/httpc (see above)!

Mint / Finch

Pre-23.3:

Connects successfully when either DST Root CA X3 or ISRG Root X1 or both are present in the CA trust store

Affected 23.3 or 24:

⚠️ As long as the DST Root CA X3 certificate is still present in the CA trust store, connections to servers with Let’s Encrypt’s long certificate chain will fail! I opened a PR for Mint that will hopefully make it into a new version before September 30th.

Once the DST Root CA X3 certificate is removed from the CA trust store, presumably after installing the next CA trust store update after September 30th, connections will succeed because Mint’s partial chain hook recognizes Let’s Encrypt’s intermediate CA certificate as having been issued by ISRG Root X1.

Update: Mint 1.4.0 is now available, which won’t fail the handshake on affected OTP versions, regardless of CA trust store contents.

Fixed 23.3 or 24:

Connects successfully when the ISRG Root X1 certificate is present in the CA trust store, regardless of whether the (expired) DST Root CA X3 its still present.

Recommendation:

If you are on 23.3 or 24.0, make sure you install the latest patch version to take advantage of the OTP’s team fix. If you are on a pre-23.3 release, make sure to jump straight to a 23.3 or 24 version that includes the fix when you do upgrade.

Tesla

Please refer to the section for the HTTP client adapter you are using with Tesla, e.g. Hackney or Mint/Finch.

Mix

Mix does not rely on TLS server certificate verification with HTTPS. Instead, commands such as mix local.hex and mix local.rebar verify the data returned by the server against a signed list of checksums. Therefore, Mix is not affected by the DST Root CA X3 expiration.

Hex

The servers that Hex normally connects to do not use a Let’s Encrypt certificate, but depending on your environment it is possible that it tries to connect to an outbound HTTP proxy or a mirror that does.

Pre-23.3:

Connects successfully when either DST Root CA X3 or ISRG Root X1 or both are present in the CA trust store

Affected 23.3 or 24:

⚠️ As long as the DST Root CA X3 certificate is still present in the CA trust store, connections to servers with Let’s Encrypt’s long certificate chain will fail!

Once the DST Root CA X3 certificate is removed from the CA trust store, as part of a Hex version update, connections will succeed because Hex’s partial chain hook recognizes Let’s Encrypt’s intermediate CA certificate as having been issued by ISRG Root X1.

Fixed 23.3 or 24:

Connects successfully when the ISRG Root X1 certificate is present in the CA trust store, regardless of whether the (expired) DST Root CA X3 its still present.

Recommendation:

If this scenario applies to you (e.g. your CI environment uses a proxy or mirror with Let’s Encrypt certificate) and you are on 23.3 or 24.0, make sure you install the latest patch version to take advantage of the OTP’s team fix.

Rebar3

Like Hex, Rebar3 does not normally connect to servers that use a Let’s Encrypt certificate, but it too may be configured to connect through a proxy, or to a mirror.

Pre-23.3:

Connects successfully when either DST Root CA X3 or ISRG Root X1 or both are present in the CA trust store

Affected 23.3 or 24:

⚠️ As long as the DST Root CA X3 certificate is still present in the CA trust store, connections to servers with Let’s Encrypt’s long certificate chain will fail!

It is possible to select a custom CA trust store by adding a ssl_cacerts_path key to the global Rebar3 config (e.g. ‘$HOME/.config/rebar3/rebar.config’). Specify a trust store that does not include the DST Root CA X3 certificate to enable connections to servers with Let’s Encrypt’s long certificate chain.

Fixed 23.3 or 24:

Connects successfully when the ISRG Root X1 certificate is present in the CA trust store, regardless of whether the (expired) DST Root CA X3 its still present.

Recommendation:

If this scenario applies to you (e.g. your CI environment uses a proxy or mirror with Let’s Encrypt certificate) and you are on 23.3 or 24.0, make sure you install the latest patch version to take advantage of the OTP’s team fix.

Back