toddholmberg
Get SSL certificate with an HTTPoison or :hackney request
Hello. I am trying to find the best way forward doing the following:
- Make an HTTPS request to an arbitrary URL.
- Make this request using a proxy.
- Be able to set the TLS/SSL protocol used for this connection.
- Be able to set the protocol-appropriate cipher used for this connection.
- Return the associated SSL certificate.
HTTPoison allows setting SSL options via its hackney foundation, but does not have return the SSL certificate with the response.
The Erlang :ssl library will allow setting the protocol and cipher for a request, but I am having trouble navigating the specifics of how to do this request via a proxy.
Has anyone had any experience making such a request and getting the associated SSL certificate? Thanks in advance for any advice.
Marked As Solved
voltone
Most HTTP clients abstract away the connection layer: they let you pass in the options to tune the connection establishment, but when they return the response they don’t include any connection information. Even Mint, which give you quite a lot of control over the lower layers, doesn’t let you extract the server certificate using a public API.
Having said that, it is possible to kind of make this work with HTTPoison. First of all, we’ll use ‘async’ mode, in which the response is delivered to a process mailbox. Then we’ll hook into the certificate verification callback and deliver the server cert (if it is valid) to the same mailbox in a custom message:
HTTPoison.get(
"https://blog.voltone.net/",
[],
stream_to: self(),
proxy: {"localhost", 8080},
hackney: [
ssl_options: [
versions: [:"tlsv1.2"],
ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
cacertfile: :certifi.cacertfile(),
depth: 3,
verify_fun: {fn
_, {:bad_cert, _} = reason, _ ->
{:fail, reason}
_,{:extension, _}, state ->
{:unknown, state}
_, :valid, state ->
{:valid, state}
peer_cert, :valid_peer, pid ->
if :public_key.pkix_verify_hostname(peer_cert, [dns_id: 'blog.voltone.net'], match_fun: :public_key.pkix_verify_hostname_match_fun(:https)) do
send(pid, {:peer_cert, peer_cert})
{:valid, pid}
else
{:fail, :hostname_check_failed}
end
end, self()}
]
]
)
Update the proxy address as needed, and remember to update the destination hostname in two places: the URI at the top, and the hostname verification function further down. Of course you could wrap this in a function that takes care of this automatically.
The result might look something like this:
iex(1)> HTTPoison.get(...)
{:ok, %HTTPoison.AsyncResponse{id: #Reference<0.1211898004.3575906308.200659>}}
iex(2)> flush
{:peer_cert,
{:OTPCertificate,
#...
}
}
%HTTPoison.AsyncStatus{
code: 200,
id: #Reference<0.1211898004.3575906308.200659>
}
%HTTPoison.AsyncHeaders{
headers: [
#...
]
}
#...
The certificate is sent as an Erlang record. You can convert it to other formats (DER, PEM) using the :public_key module.
Finally, keep in mind that not every HTTP request necessarily results in a TLS handshake, and therefore a peer certificate message. The HTTP client may keep the connection open for reuse by multiple HTTP requests, and even if a new connection is established, TLS session resumption may cause the handshake to be skipped (pass reuse_sessions: false in the ssl_options to prevent this).
Also Liked
voltone
Well, if you don’t need the HTTP response at all you can take a few shortcuts. Drop the stream_to: self() option, ignore the response from HTTPoison.get (or check only for connection errors) and then see if you got a certificate:
receive do
{:peer_cert, peer_cert} ->
{:ok, peer_cert}
after
0 ->
{:error, "No TLS handshake, or handshake failed"}
end
You can also change the method to HEAD, to avoid transferring data you’re going to throw away anyway.
At this point I’m wondering whether you need an HTTP client at all. You could just open a TCP connection to the proxy, send "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n", receive the "HTTP/1.1 200 Connection established\r\n\r\n" reply, then upgrade the connection to TLS with :ssl.connect/2 with all the necessary ssl_options. If all goes well, call :ssl.peercert/1 to get the peer certificate.
{:ok, tcp} = :gen_tcp.connect('proxy.host.name', 8080, active: false)
:gen_tcp.send(tcp, "CONNECT some.host.name:443 HTTP/1.1\r\n\r\n")
{:ok, "HTTP/1.1 200" <> _} = :gen_tcp.recv(tcp, 0)
{:ok, ssl} = :ssl.connect(tcp, [
versions: [:"tlsv1.2"],
ciphers: :ssl.cipher_suites(:default, :"tlsv1.2"),
server_name_indication: 'some.host.name',
verify: :verify_peer,
cacertfile: :certifi.cacertfile(),
depth: 3,
customize_hostname_check: [
match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
]
])
{:ok, der} = :ssl.peercert(ssl)
You should now have the peer certificate in DER binary format.
toddholmberg
toddholmberg
One more boost for @voltone : couple this answer with the recursive approach in this blog post series by @alvises and you can pattern match on just the certificate and ignore the rest of the response chunks.







