kaa.python

kaa.python

Task.await terminates GenServer because of timeout. How to fix?

My Genserver terminates after a little while, after sending a few http requests. I can’t understand the reason:

[error] GenServer MyGenServer terminating
** (stop) exited in: Task.await(%Task{owner: #PID<0.420.0>, pid: #PID<0.1054.0>, ref: #Reference<....>}, 5000)
    ** (EXIT) time out
    (elixir) lib/task.ex:416: Task.await/2
    (elixir) lib/enum.ex:966: Enum.flat_map_list/2
    (my_app123) lib/my_genserver.ex:260: MyApp.MyGenServer.do_work/1
    (my_app123) lib/my_genserver.ex:180: MyApp.MyGenServer.handle_info/2
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:683: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :tick
State: [%{var1: "fdsafdsfd", var2: "43243242"}]

A chunk of the code:

  # it's called from handle_info

  def do_work(some_data) do
    Enum.map(some_data, fn(x) ->
      Task.async(fn ->
        case HTTPoison.post(.....) do
        # ...........

Is “Task.async” causing the timeout? But why? Yes, it can take more than 5 seconds to complete, but why does it cause an exception which then terminates GenServer? How to fix it?

About await:

If the timeout is exceeded, await will exit; however, the task will continue to run. When the calling process exits, its exit signal will terminate the task if it is not trapping exits.

Most Liked

peerreynders

peerreynders

Actually exit/1 can be caught:

iex(1)> try do
...(1)>   exit("foo")
...(1)> catch 
...(1)>   :exit, _ -> :caught
...(1)> end
:caught

however your objection is still intact:

  • the timeout doesn’t have anything to do with the Task, which just keeps running - it is the await/2 process that times out
  • the termination of the await/2 process emits an exit signal to all linked processes. That exit signal cannot be “caught” - the exit signal will terminate any linked processes which aren’t trapping exits and those which do trap exits will get an :EXIT message in their mailbox.

.

  1. Kernel.exit/1 can only be caught inside the process that invokes it - if it is allowed to escape the process it turns into an exit signal - at which point it is too late to catch it anywhere.

  2. Process.exit/2 is an entirely different animal. While exit/1 terminates the process that invokes it, exit/2 sends an exit signal to the specified process usually with the intent to “tell that process to terminate”. So exit/2 cannot be caught under any circumstances (but it can be trapped - unless the reason is :kill).

.

iex(2)> try do
...(2)>   Process.exit(self(),"foo")
...(2)> catch                       
...(2)>   :exit, _ -> :caught       
...(2)> end                         
** (EXIT from #PID<0.87.0>) "foo"

.

iex(1)> Process.flag :trap_exit, true 
false
iex(2)> try do                       
...(2)>   Process.exit(self(),"foo") 
...(2)> catch                        
...(2)>   :exit, _ -> :caught        
...(2)> end                          
true
iex(3)> flush()
{:EXIT, #PID<0.103.0>, "foo"}
:ok
NobbZ

NobbZ

Task.await/1 calls exit/1 on timeout, so it will end your current process. If you do not wan’t that behaviour you will have to implement the receiving of the result and receiving of :DOWN messages on your own.

benperiton

benperiton

I’m still getting to grips with Tasks myself, but I think it’s because the Task is linked to the calling process, so if it throws an error and exits, then the calling process will also die. See https://hexdocs.pm/elixir/Task.html#module-async-and-await

You could use Task.start_link/1 if you don’t need the response, I’ve been using a Task.Supervisor for mine, so that if a Task dies, it doesn’t bring the genserver down.

Add a supervisor:

supervisor(Task.Supervisor, [[name: App.MyTaskSupervisor]])

Then can use it like:

def do_work(some_data) do
  Enum.map(some_data, fn(x) ->
    Task.Supervisor.async_nolink(App.MyTaskSupervisor, fn ->
      case HTTPoison.post(.....) do

Because I wnted to know if it was a timeout, I use yield

task = Task.Supervisor.async_nolink(
  App.MyTaskSupervisor,
  MyModule,
  :task_function,
  [args]
)

case Task.yield(task) || Task.shutdown(task) do
  {:ok, result} ->
    ack_message({:ok, %{channel: channel, tag: tag}})
  
  {:error, msg} ->
    ack_message({:error, %{error: msg, channel: channel, tag: tag, redelivered: redelivered}})

  {:exit, reason} ->
    ack_message({:error, %{error: reason, channel: channel, tag: tag, redelivered: redelivered}})

  nil ->
    ack_message({:error, %{error: "TIMEOUT", channel: channel, tag: tag, redelivered: redelivered}})
end
NobbZ

NobbZ

I’m not sure if I do understand your question correctly here. But outsourcing work from a GenServer into another process as soon as it will take longer than a couple of “cycles” is idiomatic and good practice.

This is done to keep the work the actual GenServer process has to do to a minimum so that it is able to process incomming calls and casts as fast as possible,

But as a rough concept it looks like this:

def handle_call{:do_stuff, from, status} do
  spawn(fn ->
    answer = do_stuff()
    GenServer.reply(from, answer)
  end)
  {:noreply, state}
end

Doing it this way, will remove the burden to handle any communication with that process from your GenServer. Assuming that HTTPoison does have timeouts, you can even handle them inside that spawned process and reply differently from the default when a timeout happens. Also you can try and catch there. You will still run into trouble when do_stuff or something inside of it does call exit!

peerreynders

peerreynders

Basically ditch Task.await and simply grab the results in handle_info/2. If your tasks have a habit of abnormally exiting, trap exits and process the :EXIT messages accordingly.

defmodule Charlie do
  use GenServer

  defp new_task do
    max_time = 5000
    work =
      case {(:rand.uniform max_time), (:rand.uniform max_time)} do
        {work, panic} when work < panic ->
          fn ->
            Process.sleep work
            work # return the work time as the result
          end
        {_, panic} ->
          fn ->
            Process.sleep panic
            exit(:panic) # simulate non-normal exit
          end
      end

    Task.async work
  end

  defp add_task(tasks, task),
    do: Map.put tasks, task.pid, task

  defp purge_task(tasks, pid) do
    case Map.get tasks, pid, :none do
      :none ->
        {tasks, :none}
      task ->
        Process.demonitor task.ref, [:flush]   # purge related :DOWN messages
        {(Map.delete tasks, pid), task}
    end
  end

  defp shutdown_tasks(tasks) do
    tasks
    |> Map.values()
    |> Enum.each(&(Task.shutdown &1, :brutal_kill))
  end

  ## callbacks
  def handle_info(:next_task, {tasks, _ref}) do
    # time to create another task
    new_tasks =
      cond do
        (length (Map.keys tasks)) < 50 ->
          add_task tasks, new_task()
        true ->
          tasks
      end

    timer_ref = Process.send_after self(), :next_task, 500
    {:noreply, {new_tasks, timer_ref}}
  end
  def handle_info({ref, result}, state) when is_reference ref do
    # handle Task completion
    Process.demonitor ref, [:flush]   # purge related :DOWN messages ASAP

    IO.puts "Task #{inspect ref} completed with result #{result}"

    {:noreply, state} # remove task when :EXIT :normal is processed
  end
  def handle_info({:EXIT, from, reason}, {tasks, timer_ref} = state) do
    # Handle exit signal not handled by the behaviour (i.e. signal from parent process)
    case purge_task tasks, from  do
      {_, :none} ->
        IO.puts "Unknown :EXIT #{inspect reason}"
        case reason do
          :normal ->
            {:noreply, state}
          _ ->
            {:stop, reason, state}  # Follow protocol: unknown non-normal exit signal - time to terminate
        end
      {new_tasks, task} ->
        case reason do
          :normal ->
            :ok
          _ -> # The task panicked
            IO.puts "Task #{inspect task.ref} panicked: #{inspect reason}"
        end
        {:noreply, {new_tasks, timer_ref}}
    end
  end

  def init(_args) do
    Process.flag(:trap_exit, true)
    Kernel.send self(), :next_task # start spinning off tasks
    {:ok, {%{}, :none}}
  end

  def terminate(_reason, {tasks, timer_ref}) do
    IO.puts "Terminating"
    cond do
      is_reference timer_ref ->
        Process.cancel_timer timer_ref
      true ->
        0 # no timer, therefore no time left
    end

    shutdown_tasks tasks
    :ok
  end

  ## public interface

  def start_link do
    GenServer.start_link(__MODULE__, [])
  end

  def stop(pid) do
    GenServer.stop pid
  end

  # NOTE:
  # https://hexdocs.pm/elixir/Task.html#await/2-compatibility-with-otp-behaviours
  # It is not recommended to await a long-running task inside an OTP behaviour such as GenServer.
  # Instead, you should match on the message coming from a task inside your
  # GenServer.handle_info/2 callback.
  #
end

.

$iex -S mix
iex(1)> Process.flag :trap_exit, true 
false
iex(2)> {:ok,pid} = Charlie.start_link
{:ok, #PID<0.127.0>}
Task #Reference<0.0.6.653> panicked: :panic
Task #Reference<0.0.6.651> completed with result 1306
Task #Reference<0.0.6.647> panicked: :panic
Task #Reference<0.0.6.658> panicked: :panic
Task #Reference<0.0.5.605> panicked: :panic
Task #Reference<0.0.6.649> panicked: :panic
Task #Reference<0.0.6.661> completed with result 464
Task #Reference<0.0.6.668> panicked: :panic
Task #Reference<0.0.6.677> panicked: :panic
Task #Reference<0.0.6.673> panicked: :panic
Task #Reference<0.0.6.682> completed with result 957
Task #Reference<0.0.6.684> completed with result 665
Task #Reference<0.0.6.675> panicked: :panic
iex(3)> Process.exit pid, :whatever
Terminating
true
iex(4)>

Where Next?

Popular in Questions Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
makeitrein
Hey all, just started picking up Elixir last week and am writing a scraper as a learning project. Baby step #1 is extracting the number ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
wernerlaude
In AR this is so simple @articles = current_user.articles How to do in Ecto? def index(conn, _params) do current_user = conn.assig...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

Other popular topics Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement