toddholmberg

toddholmberg

Get SSL certificate with an HTTPoison or :hackney request

Hello. I am trying to find the best way forward doing the following:

  1. Make an HTTPS request to an arbitrary URL.
  2. Make this request using a proxy.
  3. Be able to set the TLS/SSL protocol used for this connection.
  4. Be able to set the protocol-appropriate cipher used for this connection.
  5. Return the associated SSL certificate.

HTTPoison allows setting SSL options via its hackney foundation, but does not have return the SSL certificate with the response.

The Erlang :ssl library will allow setting the protocol and cipher for a request, but I am having trouble navigating the specifics of how to do this request via a proxy.

Has anyone had any experience making such a request and getting the associated SSL certificate? Thanks in advance for any advice.

Marked As Solved

voltone

voltone

Most HTTP clients abstract away the connection layer: they let you pass in the options to tune the connection establishment, but when they return the response they don’t include any connection information. Even Mint, which give you quite a lot of control over the lower layers, doesn’t let you extract the server certificate using a public API.

Having said that, it is possible to kind of make this work with HTTPoison. First of all, we’ll use ‘async’ mode, in which the response is delivered to a process mailbox. Then we’ll hook into the certificate verification callback and deliver the server cert (if it is valid) to the same mailbox in a custom message:

HTTPoison.get(
  "https://blog.voltone.net/",
  [],
  stream_to: self(),
  proxy: {"localhost", 8080},
  hackney: [
    ssl_options: [
      versions: [:"tlsv1.2"],
      ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
      cacertfile: :certifi.cacertfile(),
      depth: 3,
      verify_fun: {fn
        _, {:bad_cert, _} = reason, _ ->
          {:fail, reason}

        _,{:extension, _}, state ->
          {:unknown, state}

        _, :valid, state ->
          {:valid, state}

        peer_cert, :valid_peer, pid ->
          if :public_key.pkix_verify_hostname(peer_cert, [dns_id: 'blog.voltone.net'], match_fun: :public_key.pkix_verify_hostname_match_fun(:https)) do            
            send(pid, {:peer_cert, peer_cert})
            {:valid, pid}
          else
            {:fail, :hostname_check_failed}
          end
      end, self()}
    ]
  ]
)

Update the proxy address as needed, and remember to update the destination hostname in two places: the URI at the top, and the hostname verification function further down. Of course you could wrap this in a function that takes care of this automatically.

The result might look something like this:

iex(1)> HTTPoison.get(...)
{:ok, %HTTPoison.AsyncResponse{id: #Reference<0.1211898004.3575906308.200659>}}
iex(2)> flush
{:peer_cert,
 {:OTPCertificate,
  #...
 }
}
%HTTPoison.AsyncStatus{
  code: 200,
  id: #Reference<0.1211898004.3575906308.200659>
}
%HTTPoison.AsyncHeaders{
  headers: [
    #...
  ]
}
#...

The certificate is sent as an Erlang record. You can convert it to other formats (DER, PEM) using the :public_key module.

Finally, keep in mind that not every HTTP request necessarily results in a TLS handshake, and therefore a peer certificate message. The HTTP client may keep the connection open for reuse by multiple HTTP requests, and even if a new connection is established, TLS session resumption may cause the handshake to be skipped (pass reuse_sessions: false in the ssl_options to prevent this).

Also Liked

voltone

voltone

Well, if you don’t need the HTTP response at all you can take a few shortcuts. Drop the stream_to: self() option, ignore the response from HTTPoison.get (or check only for connection errors) and then see if you got a certificate:

receive do
  {:peer_cert, peer_cert} ->
    {:ok, peer_cert}
after
  0 ->
    {:error, "No TLS handshake, or handshake failed"}
end

You can also change the method to HEAD, to avoid transferring data you’re going to throw away anyway.

At this point I’m wondering whether you need an HTTP client at all. You could just open a TCP connection to the proxy, send "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n", receive the "HTTP/1.1 200 Connection established\r\n\r\n" reply, then upgrade the connection to TLS with :ssl.connect/2 with all the necessary ssl_options. If all goes well, call :ssl.peercert/1 to get the peer certificate.

{:ok, tcp} = :gen_tcp.connect('proxy.host.name', 8080, active: false)
:gen_tcp.send(tcp, "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n")
{:ok, "HTTP/1.1 200" <> _} = :gen_tcp.recv(tcp, 0)
{:ok, ssl} = :ssl.connect(tcp, [
  versions: [:"tlsv1.2"],
  ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
  server_name_indication: 'some.host.name',
  verify: :verify_peer,
  cacertfile: :certifi.cacertfile(),
  depth: 3,
  customize_hostname_check: [
    match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
  ]
])
{:ok, der} = :ssl.peercert(ssl)

You should now have the peer certificate in DER binary format.

toddholmberg

toddholmberg

Thank you, @voltone! This is exactly the help I needed. I will also be using your excellent X509 package to process the resulting certificate.

toddholmberg

toddholmberg

One more boost for @voltone : couple this answer with the recursive approach in this blog post series by @alvises and you can pattern match on just the certificate and ignore the rest of the response chunks.

Where Next?

Popular in Questions Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

Other popular topics Top

vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New

We're in Beta

About us Mission Statement