Fl4m3Ph03n1x
How to throttle HTTP requests?
Background
I have a small terminal application that does HTTP requests to a website. However, I am making too many requests per second and I am drowning the receiver.
Solution
The obvious solution is to throttle the rate at which I perform requests.
Since I am using HTTPoison I thought about reading the docs to see if this was a feature, but I didn’t find anything.
The other solution is to find a community library that does this for me. This library would have to not use any external tools (like redis) and throttle out-going requests. After some investigation I didn’t really find anything in the community.
Surely this is something out there and I missed it.
Questions
What libraries do you use when you need to throttle out-going requests?
Marked As Solved
Fl4m3Ph03n1x
As a final remark to this thread, I just wanted to say I eventually went with the @cmkarlsson’s suggestion and used the jobs library.
However, I also read @akoutmos’s blog, and I can say that although it is not a reading for beginners, I very much enjoyed it and I can thoroughly recommend it to anyone wanting to learn more about throttlers.
Also Liked
cmkarlsson
There is an erlang library for load regulation that I have used in the past.
At the simplest case you can start a job queue which will only allow a specific amount of requests per second. It doesn’t use any external dependencies.
At its simplest:
> jobs.add_queue(:myqueue, [{:standard_rate, 10}])
> jobs.run(:myqueue, fn -> :httpc.request('https://www.elixirforum.com') end)
Runs the jobs at a rate of 10 per second at most
akoutmos
I actually wrote a blog post about this exact problem and go through the process of implementing token bucket and leaky bucket rate limiters using a simple GenServer and a Task.Supervisor: https://akoutmos.com/post/rate-limiting-with-genservers/
I’d imagine that you want to persist your queue in the case that the application goes down. You can add that in pretty easily to the code snippets I walk through by implementing the terminate/2 callback.
Hope that helps!







