Fl4m3Ph03n1x

Fl4m3Ph03n1x

Not getting response body with :gun

Background

Some time ago someone in this wonderful community suggested I used gun as an HTTP client, given that I was having severe issues with HTTPoison and later on had them with HTTPotion as well (they didn’t scale well enough).

Code

To fix it, we moved our solution to use the asynchronous HTTP client mentioned above: gun.
gun is an erlang library that communicates with a GenServer via events, calls and casts. To use gun I have therefore built a primitive GenServer client that prints to the console everything it receives:

defmodule ConnPoolWorker do
  use GenServer
  alias ProcessRegistry
  alias :gun, as: Gun

  @url_domain 'google.com'
  @https_port 443

  def start_link({worker_id}) do
    IO.puts("Starting worker #{worker_id}")

    GenServer.start_link(
      __MODULE__,
      nil,
      name: via_tuple(worker_id)
    )
  end

  ## Public API

  #makes a GET request via gun
  def fire(worker_id, url) do
    GenServer.cast(via_tuple(worker_id), {:fire, url})
  end
  
   ## Implementation 
   defp via_tuple(worker_id) do
    ProcessRegistry.via_tuple({__MODULE__, worker_id})
  end

  @impl GenServer
  def init(_args) do
    {:ok, conn_pid} = Gun.open(@url_domain, @https_port)
    {:ok, _protocol} = Gun.await_up(conn_pid)
    {:ok, conn_pid}
  end

  @impl GenServer
  def handle_cast({:fire, url}, conn_pid) do
    Gun.get(conn_pid, url)
    {:noreply, conn_pid}
  end

  # handle_info everything else
  @impl GenServer
  def handle_info(msg, state) do
    IO.puts("MSG: #{inspect msg}")
    {:noreply, state}
  end
end

This client is registered in the Registry using via_tuples, but I don’t think that is majorly important for now.

Problem

This code works. It opens a connections, waits for the connection to be up, and if you invoke fire to make a request (get it? because the library is called gun? :smiley: ) you get a :gun_response event that handle_info picks up.

However, that’s precisely the issue. It’s the only event the process ever picks up. This process gets no other events like :gun_data or :gun_trailers, even though the documentation says it should.

The only thing I get every now and then is a :gun_down (connection down) followed by a :gun_up (connection up) which is normal:

MSG: {:gun_response, #PID<0.214.0>, #Reference<0.1474087065.2991849473.15656>, :fin, 302, [{"server", "nginx"}, {"date", "Wed, 20 Feb 2019 11:55:14 GMT"}, {"content-length", "0"}, {"connection", "keep-alive"}, {"location", "http://www.sapo.pt/noticias/"}, {"strict-transport-security", "max-age=31536000"}, {"x-content-type-options", "nosniff"}, {"content-security-policy", "upgrade-insecure-requests; block-all-mixed-content"}, {"x-xss-protection", "1; mode=block"}, {"referrer-policy", "origin-when-cross-origin"}]}
MSG: {:gun_down, #PID<0.221.0>, :http, :closed, [], []}
MSG: {:gun_down, #PID<0.224.0>, :http, :closed, [], []}

Question

  1. Am I doing something wrong with my GenServer client?
  2. Is this working as intended or am I missing something?
  3. How can I get the rest of the data from the request?

Marked As Solved

Fl4m3Ph03n1x

Fl4m3Ph03n1x

I found the issue. Turns out I was getting the full response:

{:gun_response, #PID<0.214.0>, #Reference<0.1474087065.2991849473.15656>, :fin, ....}

The last argument :fin means this is the only response I am supposed to get. So, both :gun and my server are working as intended as I am getting a redirection code 302.

Also Liked

amnu3387

amnu3387

I think in your case continue could work fine because you’re instantiating independent genservers that then receive messages to do their work, although doing the bootstrap on init is probably better in this particular case (I say this not knowing the remaining of the system).

Still, to understand how {:continue ...} works you need to look into the process message box queue erlang uses.

Say you have GenServer “Server” and now, concurrently from any part of your system you have multiple processes sending “Server” a message

p1 ! Server :message_a
p2 ! Server :message_b
p3 ! Server :message_c

Now the BEAM guarantees that the messages are delivered in order to the process “Server” (caveats apply if processes are spread on nodes and not on a single one)

So “Server” has in its mailbox: [:message_a, :message_b, :message_c]

And it handles one by one, handle_info(any, state), so the first handle matching will be called with msg a, then b, then c.

If for instance you have a handle that continues, say for :message_b

handle_info(:message_b, state), do: {:noreply, state, {:continue, :recalc}}

What happens is, back to the same 3 sent messages

[:message_a, :message_b, :message_c]
Processed A
[:message_b, :message_c]
Processed B
:continue
[:continue_recalc, :message_c]
Processed handle_continue :recalc
[:message_c]
Processed C
[]

So it basically hijacks the message box and puts on its first item whatever is the continuation handle, and that is guaranteed to run before any other message is processed. That continuation handle can itself continue to something else, and if it did, that would be guaranteed to be processed as well before any other message.

In your case, let’s say you start ConnPoolWorker.start_link({1}) on your supervision tree

def init(_) do
   {:ok, :not_ready, {:continue, :open}}
end

Here it returns ok, the supervisor receives the :ok, and continues starting the remaining tree, somewhere else on your app down the line, some other process are started and they start sending fire requests, say 5000 , the message box now has 5000 requests, but you’re guaranteed that before trying to process any of those, the continue handle will be executed.

def handle_continue(:open, _) do
   {:ok, conn_pid} = Gun.open(@url_domain, @https_port)
   {:ok, _protocol} = Gun.await_up(conn_pid)
   {:noreply, conn_pid}
end

So only when this handle continue executes are the remaining messages processed.

Of course if this handle_continue fails (say Gun.open errors out), then the genserver will crash, and those messages already sent and waiting on the process’ message box will be lost, whereas if you start it as you do in init, and there’s no way for the processes down the line to be started and start firing requests unless the init is successful, it might be better to simply do the bootstrap on init (as you’re doing) and prevent the app from even going further if the gun connections can’t be open.

But if you don’t loose the “targets” until the requests are successfully done, meaning if the app/processes crashes when they restart you’ll be able to fire the same requests because they haven’t been completed, there wouldn’t be a problem in using :continue.

One example where you probably don’t want to use :continue is when your genserver instantiates things that won’t be interacted by its own message box (like say, you’re populating a big ETS table that is bootstrapped from a DB) and this ETS is going to be read directly by other processes. In this case using :continue to populate the ETS table could mean that other processes start trying to access the ETS before it’s fully populated/even created, while populating it on init would guarantee that it’s ready before the Supervision Tree continues starting the other bits of your system that might interact with it. But even in this case, it might be a reasonable tradeoff, it depends.

shanesveller

shanesveller

I would strongly encourage moving the connection building out of your init into a handle_continue if you will be spooling off a lot of these in rapid succession, which feels likely for an HTTP client wrapper. init is blocking until it returns, which means anything calling start_link on this module is blocking too.

Fl4m3Ph03n1x

Fl4m3Ph03n1x

An excellent response, that I have bookmarked for future reference !

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
makeitrein
Hey all, just started picking up Elixir last week and am writing a scraper as a learning project. Baby step #1 is extracting the number ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New

We're in Beta

About us Mission Statement