The great HTTPS client shoot-out

Posted 2016-11-05 08:03:50.000000

Let me start off by apologising for the click-bait title. A better title would be: “On the security posture of Elixir HTTPS clients”. It’s a topic I covered in my ElixirConf EU talk, but it recently came up again.

The goal of this post is not to compare features, performance or usability, or even to choose a ‘most secure’ client. Instead it tries to establish what it takes to use these clients to connect securely to a server, given an HTTPS URL.

In many Elixir applications, an HTTP client is used to connect to 3rd party API servers a such as AWS, Facebook or Google. Requests often contain sensitive data, such as API keys or personal information, that must be protected against eavesdropping and man-in-the-middle attacks. Developers might expect an HTTP client that accepts HTTPS URLs to take care of these things, but things aren’t always that simple, as we shall see…

Please meet the contenders:

Setting up

If you want to follow along, create a new project using mix new, edit your mix.exs file to list the package and application dependencies as shown below and then run mix deps.get.

defmodule HTTPClients.Mixfile do
  # ...snip...

  def application do
    [applications: [:logger, :httpoison, :httpotion, :simplehttp, :inets, :ssl]]
  end

  # ...snip...
  defp deps do
    [
      {:httpoison, ">= 0.0.0"},
      {:httpotion, ">= 0.0.0"},
      {:simplehttp, ">= 0.0.0"},
    ]
  end
end

You may also want to have a set of trusted CA certificates available. You can get one here:

mkdir priv
wget -q https://curl.haxx.se/ca/cacert.pem -O priv/cacert.pem

Now run iex -S mix and you’re ready to start interactively exploring the HTTP client APIs.

Important note: most HTTP clients include a connection pool, and Erlang’s ssl module supports session resumption, which can lead to unexpected behaviour while experimenting with different TLS options. If changes don’t seem to have any effect from one request to the next, restart your application or iex session to make sure a fresh TLS handshake will take place.

Test endpoints

We will be using this blog as a test server for the various scenarios:

UPDATE: Please note that the test endpoints are no longer available!

In a follow-up post I will explain how you can build your own TLS test server using Phoenix in just a few lines of code, and how to use the OpenSSL CLI to debug handshake issues.

Now then, let’s get started with our first client…

HTTPoison (hackey)

Version used: 0.9.2 (with hackney 1.6.3)

iex(1)> HTTPoison.get("https://blog.voltone.net/")
{:ok, %HTTPoison.Response{...}}
iex(2)> HTTPoison.get("https://selfsigned.voltone.net/")
{:error, %HTTPoison.Error{id: nil, reason: {:tls_alert, 'bad certificate'}}}
iex(3)> HTTPoison.get("https://mismatch.voltone.net/")  
{:error, %HTTPoison.Error{id: nil, reason: {:tls_alert, 'handshake failure'}}}

Perfect! Thanks to hackney’s certifi and ssl_verify_fun dependencies, along with sensible defaults, HTTPoison correctly detects the certificate issues. Depending on your Erlang/OTP release and your logging configuration you may see log messages containing a slightly more specific reason for the failure.

What if we wanted to use HTTPoison to connect to a private server with a self-signed certificate? No problem, just override the ssl options, setting :verify to :verify_none. This will disable server certificate verification, meaning we loose the MitM protection, but TLS still provides protection against ‘casual’ eavesdroppers:

iex(4)> HTTPoison.get("https://selfsigned.voltone.net/", [], ssl: [verify: :verify_none])
{:ok, %HTTPoison.Response{...}}

Remember to check for updates to the certifi package on a regular basis: if CA certificates have been added to or removed from the trust store, you’ll likely want your application to pick up those changes:

$ mix hex.outdated --all
Dependency      Current  Latest  Requirement  
certifi         0.6.0    0.7.0   0.7.0
...
$ mix deps.update certifi
Running dependency resolution
Dependency resolution completed
  certifi: 0.7.0
* Updating certifi (Hex package)
  Checking package (https://repo.hex.pm/tarballs/certifi-0.7.0.tar)
  Using locally cached package
$

HTTPotion (ibrowse)

Version used: 3.0.2 (with ibrowse 4.2.2)

iex(1)> HTTPotion.get("https://blog.voltone.net/")
%HTTPotion.Response{status_code: 200, ...}
iex(2)> HTTPotion.get("https://selfsigned.voltone.net/")
%HTTPotion.Response{status_code: 200, ...}
iex(3)> HTTPotion.get("https://mismatch.voltone.net/")  
%HTTPotion.Response{status_code: 200, ...}

It’s clear that HTTPotion and ibrowse do not perform server certificate verification by default. That’s because neither a CA trust store nor a hostname verification function is included in these libraries. That’s fair enough, but it might have been better to set verify to :verify_peer by default:

iex(4)> HTTPotion.get("https://blog.voltone.net/", ibrowse: [ssl_options: [verify: :verify_peer]])           
%HTTPotion.ErrorResponse{message: "{:options, {:cacertfile, []}}"}

Now any HTTPS connections fail, before the TLS handshake is even initiated. This forces the developer to choose between explicitly overriding the verify option or adding the necessary options to make HTTPS work securely. You can make this HTTPotion’s default behaviour in your own projects by adding the following lines you your config/config.exs file:

config :httpotion, :default_ibrowse,
  ssl_options: [verify: :verify_peer]

After restarting iex you’ll find that HTTPotion will not connect to HTTPS URLs unless you explicitly pass the :verify_none value:

iex(1)> HTTPotion.get("https://blog.voltone.net/")
%HTTPotion.ErrorResponse{message: "{:options, {:cacertfile, []}}"}
iex(2)> HTTPotion.get("https://blog.voltone.net/", ibrowse: [ssl_options: [verify: :verify_none]])
%HTTPotion.Response{status_code: 200, ...}

Okay, that prevents some nasty surprises, but how do we enable proper server certificate verification?

First of all, we need to tell the ssl module which CA certificates we trust. One way to do that is to pass in the cacertfile option. Choose whichever method you prefer for getting the full path to the file, just remember that Erlang expects to get paths as character lists, not strings (binaries):

iex(1)> cacertfile = :code.priv_dir(:http_clients) ++ '/cacert.pem'
...
iex(2)> cacertfile = :http_clients |> Application.app_dir(["priv", "cacert.pem"]) |> to_charlist
...
iex(3)> HTTPotion.get("https://blog.voltone.net/", ibrowse: [ssl_options: [verify: :verify_peer, cacertfile: cacertfile]])
%HTTPotion.Response{status_code: 200, ...}
iex(4)> HTTPotion.get("https://selfsigned.voltone.net/", ibrowse: [ssl_options: [verify: :verify_peer, cacertfile: cacertfile]])
%HTTPotion.ErrorResponse{message: "{:tls_alert, 'bad certificate'}"}

Note that HTTPotion does not deep-merge the ssl_options from your application’s config/config.exs file with the parameters passed in, so remember to include all desired ssl_options, including :verify_peer.

This stops HTTPotion from silently accepting any self-signed certificate, but it does not enable hostname verification. An attacker can still pull off a man-in-the-middle attack by simply presenting any trusted server certificate, for a hostname under the attacker’s control, and HTTPotion will not detect the mismatch:

iex(5)> HTTPotion.get("https://mismatch.voltone.net/", ibrowse: [ssl_options: [cacertfile: cacertfile, verify: :verify_peer]])  
%HTTPotion.Response{status_code: 200, ...}

What we need is a callback implementation that we can pass into the verify_fun option. We could write our own, based on RFC 6125, but we might as well re-use the implementation used by HTTPoison/hackney, in the ssl_verify_fun package. It is already available in our sample application, as a transitive dependency, and in your own applications you can simply add it as a direct dependency.

Here’s how we would use it:

iex(6)> HTTPotion.get("https://blog.voltone.net/", ibrowse: [ssl_options: [cacertfile: cacertfile, verify: :verify_peer, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: 'blog.voltone.net']}]])    
%HTTPotion.Response{status_code: 200, ...}
iex(7) HTTPotion.get("https://mismatch.voltone.net/", ibrowse: [ssl_options: [cacertfile: cacertfile, verify: :verify_peer, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: 'mismatch.voltone.net']}]])
%HTTPotion.ErrorResponse{message: "{:tls_alert, 'handshake failure'}"}

Note that it is necessary to pass in the expected hostname explicitly (as a charlist): it is not extracted from the URL passed into HTTPotion/ibrowse, since they are not aware of the verify_fun callback, nor by the Erlang ssl module since it never sees the URL. Because of this requirement it is not possible to configure the verify_fun option globally in the :httpotion application configuration, so all of the ssl_options shown above must be included in all calls to the HTTPotion API.

Finally, we could replace the CA trust store in our application’s priv directory with the certifi package. Assuming the package is available as a dependency, as it is in our test project, we can replace the cacertfile option with the cacerts option, passing in the certificate list:

iex(7)> HTTPotion.get("https://blog.voltone.net/", ibrowse: [ssl_options: [cacerts: :certifi.cacerts(), verify: :verify_peer, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: 'blog.voltone.net']}]])
%HTTPotion.Response{status_code: 200, ...}

SimpleHttp

Version used: 0.4.0

iex(1)> SimpleHttp.get("https://blog.voltone.net/")
{:ok, %SimpleHttp.Response{...}
iex(2)> SimpleHttp.get("https://selfsigned.voltone.net/")
{:ok, %SimpleHttp.Response{...}
iex(3)> SimpleHttp.get("https://mismatch.voltone.net/")  
{:ok, %SimpleHttp.Response{...}

Just like HTTPotion/ibrowse, SimpleHttp does not perform TLS server certificate verification. Unfortunately it seems SimpleHttp does not (currently) allow ssl options to be specified, so it is not possible to make HTTPS work securely. For this reason I cannot recommend the use of this client at this time. Instead, you should consider using the httpc client directly as described below.

httpc

Version used: inets 6.3.3, included in Erlang/OTP 19.1 (erts-8.1)

httpc is the HTTP client included in Erlang/OTP as part of the inets application. It might be an attractive option if you wish to keep your application’s dependencies to a minimum. There is a good case to be made for such a policy. However, httpc has a few peculiarities of its own, so your mileage may vary.

iex(1)> :httpc.request('https://blog.voltone.net')
{:ok, {{'HTTP/1.1', 200, 'OK'}, ...}}
iex(2)> :httpc.request('https://selfsigned.voltone.net')
{:ok, {{'HTTP/1.1', 200, 'OK'}, ...}}
iex(3)> :httpc.request('https://mismatch.voltone.net')  
{:ok, {{'HTTP/1.1', 200, 'OK'}, ...}}

A familiar pattern: no server certificate verification out-of-the-box. Let’s first override the verify option:

iex(4)> :httpc.set_options(socket_opts: [verify: :verify_peer])
:ok
iex(5)> :httpc.request('https://blog.voltone.net')             
{:error,
 {:failed_connect,
  [{:to_address, {'blog.voltone.net', 443}},
   {:inet, [:inet, {:verify, :verify_peer}], {:options, {:cacertfile, []}}}]}}

As before, this prevents the insecure use of HTTPS URLs, instead of silently ignoring the server’s certificate. You may want to add the set_options call to your application initialisation when using httpc.

The next step is to actually verify the server’s certificate chain against a trust store:

iex(4)> cacertfile = :code.priv_dir(:http_clients) ++ '/cacert.pem'
...
iex(5)> :httpc.set_options(socket_opts: [verify: :verify_peer, cacertfile: cacertfile])
:ok
iex(6)> :httpc.request('https://blog.voltone.net')
{:ok, {{'HTTP/1.1', 200, 'OK'}, ...}}
iex(7)> :httpc.request('https://selfsigned.voltone.net')
{:error,
 {:failed_connect,
  [{:to_address, {'selfsigned.voltone.net', 443}},
   {:inet,
    [:inet, {:verify, :verify_peer},
     {:cacertfile,
      '[...]/http_clients/_build/dev/lib/http_clients/priv/cacert.pem'}],
    {:tls_alert, 'bad certificate'}}]}}

Adding hostname verification with the ssl_verify_fun package is very similar to what we did for HTTPotion: in order to pass the expected hostname value as part of the verify_fun option, we need to pass in ssl options as part of the request. With httpc this means using the more elaborate request/4 API:

iex(7)> :httpc.request(:get, {'https://mismatch.voltone.net', []}, [ssl: [verify: :verify_peer, cacertfile: cacertfile, verify_fun: {&:ssl_verify_hostname.verify_fun/3, [check_hostname: 'mismatch.voltone.net']}]], [])
{:error,
 {:failed_connect,
  [{:to_address, {'mismatch.voltone.net', 443}},
   {:inet, [:inet], {:tls_alert, 'handshake failure'}}]}}

And there you have it. I still haven’t had a chance to add comments to the blog engine, sorry. Instead, feel free to get in touch via Twitter or share your thoughts on this post in this Elixir Forum thread.

Back