toddholmberg

toddholmberg

Get SSL certificate with an HTTPoison or :hackney request

Hello. I am trying to find the best way forward doing the following:

  1. Make an HTTPS request to an arbitrary URL.
  2. Make this request using a proxy.
  3. Be able to set the TLS/SSL protocol used for this connection.
  4. Be able to set the protocol-appropriate cipher used for this connection.
  5. Return the associated SSL certificate.

HTTPoison allows setting SSL options via its hackney foundation, but does not have return the SSL certificate with the response.

The Erlang :ssl library will allow setting the protocol and cipher for a request, but I am having trouble navigating the specifics of how to do this request via a proxy.

Has anyone had any experience making such a request and getting the associated SSL certificate? Thanks in advance for any advice.

Marked As Solved

voltone

voltone

Most HTTP clients abstract away the connection layer: they let you pass in the options to tune the connection establishment, but when they return the response they don’t include any connection information. Even Mint, which give you quite a lot of control over the lower layers, doesn’t let you extract the server certificate using a public API.

Having said that, it is possible to kind of make this work with HTTPoison. First of all, we’ll use ‘async’ mode, in which the response is delivered to a process mailbox. Then we’ll hook into the certificate verification callback and deliver the server cert (if it is valid) to the same mailbox in a custom message:

HTTPoison.get(
  "https://blog.voltone.net/",
  [],
  stream_to: self(),
  proxy: {"localhost", 8080},
  hackney: [
    ssl_options: [
      versions: [:"tlsv1.2"],
      ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
      cacertfile: :certifi.cacertfile(),
      depth: 3,
      verify_fun: {fn
        _, {:bad_cert, _} = reason, _ ->
          {:fail, reason}

        _,{:extension, _}, state ->
          {:unknown, state}

        _, :valid, state ->
          {:valid, state}

        peer_cert, :valid_peer, pid ->
          if :public_key.pkix_verify_hostname(peer_cert, [dns_id: 'blog.voltone.net'], match_fun: :public_key.pkix_verify_hostname_match_fun(:https)) do            
            send(pid, {:peer_cert, peer_cert})
            {:valid, pid}
          else
            {:fail, :hostname_check_failed}
          end
      end, self()}
    ]
  ]
)

Update the proxy address as needed, and remember to update the destination hostname in two places: the URI at the top, and the hostname verification function further down. Of course you could wrap this in a function that takes care of this automatically.

The result might look something like this:

iex(1)> HTTPoison.get(...)
{:ok, %HTTPoison.AsyncResponse{id: #Reference<0.1211898004.3575906308.200659>}}
iex(2)> flush
{:peer_cert,
 {:OTPCertificate,
  #...
 }
}
%HTTPoison.AsyncStatus{
  code: 200,
  id: #Reference<0.1211898004.3575906308.200659>
}
%HTTPoison.AsyncHeaders{
  headers: [
    #...
  ]
}
#...

The certificate is sent as an Erlang record. You can convert it to other formats (DER, PEM) using the :public_key module.

Finally, keep in mind that not every HTTP request necessarily results in a TLS handshake, and therefore a peer certificate message. The HTTP client may keep the connection open for reuse by multiple HTTP requests, and even if a new connection is established, TLS session resumption may cause the handshake to be skipped (pass reuse_sessions: false in the ssl_options to prevent this).

Also Liked

voltone

voltone

Well, if you don’t need the HTTP response at all you can take a few shortcuts. Drop the stream_to: self() option, ignore the response from HTTPoison.get (or check only for connection errors) and then see if you got a certificate:

receive do
  {:peer_cert, peer_cert} ->
    {:ok, peer_cert}
after
  0 ->
    {:error, "No TLS handshake, or handshake failed"}
end

You can also change the method to HEAD, to avoid transferring data you’re going to throw away anyway.

At this point I’m wondering whether you need an HTTP client at all. You could just open a TCP connection to the proxy, send "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n", receive the "HTTP/1.1 200 Connection established\r\n\r\n" reply, then upgrade the connection to TLS with :ssl.connect/2 with all the necessary ssl_options. If all goes well, call :ssl.peercert/1 to get the peer certificate.

{:ok, tcp} = :gen_tcp.connect('proxy.host.name', 8080, active: false)
:gen_tcp.send(tcp, "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n")
{:ok, "HTTP/1.1 200" <> _} = :gen_tcp.recv(tcp, 0)
{:ok, ssl} = :ssl.connect(tcp, [
  versions: [:"tlsv1.2"],
  ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
  server_name_indication: 'some.host.name',
  verify: :verify_peer,
  cacertfile: :certifi.cacertfile(),
  depth: 3,
  customize_hostname_check: [
    match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
  ]
])
{:ok, der} = :ssl.peercert(ssl)

You should now have the peer certificate in DER binary format.

toddholmberg

toddholmberg

Thank you, @voltone! This is exactly the help I needed. I will also be using your excellent X509 package to process the resulting certificate.

toddholmberg

toddholmberg

One more boost for @voltone : couple this answer with the recursive approach in this blog post series by @alvises and you can pattern match on just the certificate and ignore the rest of the response chunks.

Where Next?

Popular in Questions Top

stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New

We're in Beta

About us Mission Statement