Fl4m3Ph03n1x

Fl4m3Ph03n1x

How to throttle HTTP requests?

Background

I have a small terminal application that does HTTP requests to a website. However, I am making too many requests per second and I am drowning the receiver.

Solution

The obvious solution is to throttle the rate at which I perform requests.
Since I am using HTTPoison I thought about reading the docs to see if this was a feature, but I didn’t find anything.

The other solution is to find a community library that does this for me. This library would have to not use any external tools (like redis) and throttle out-going requests. After some investigation I didn’t really find anything in the community.

Surely this is something out there and I missed it.

Questions

What libraries do you use when you need to throttle out-going requests?

Marked As Solved

Fl4m3Ph03n1x

Fl4m3Ph03n1x

As a final remark to this thread, I just wanted to say I eventually went with the @cmkarlsson’s suggestion and used the jobs library.

However, I also read @akoutmos’s blog, and I can say that although it is not a reading for beginners, I very much enjoyed it and I can thoroughly recommend it to anyone wanting to learn more about throttlers.

Also Liked

cmkarlsson

cmkarlsson

There is an erlang library for load regulation that I have used in the past.

At the simplest case you can start a job queue which will only allow a specific amount of requests per second. It doesn’t use any external dependencies.

At its simplest:

> jobs.add_queue(:myqueue, [{:standard_rate, 10}])
> jobs.run(:myqueue, fn -> :httpc.request('https://www.elixirforum.com') end)

Runs the jobs at a rate of 10 per second at most

akoutmos

akoutmos

Author of Build a Weather Station with Elixir and Nerves

I actually wrote a blog post about this exact problem and go through the process of implementing token bucket and leaky bucket rate limiters using a simple GenServer and a Task.Supervisor: https://akoutmos.com/post/rate-limiting-with-genservers/

I’d imagine that you want to persist your queue in the case that the application goes down. You can add that in pretty easily to the code snippets I walk through by implementing the terminate/2 callback.

Hope that helps!

Where Next?

Popular in Questions Top

vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

We're in Beta

About us Mission Statement