artem

artem

What's good way to have limits for calling external APIs on a **monthly** or daily level?

Hi all

Need for exactly monthly limiting

I am learning Elixir (and Phoenix and LiveView) by creating a simple app that would fetch weather from external service and “creatively” present it (think “today is a bit warmer that yesterday” compared to “today is +17”).

At some point I’ll make the app public. Certainly for a hobby project I don’t want to pay much for the weather API and have to stay within free or cheap pricing tiers that is my Elixir backend should call external API no more than e.g. 1000 times per calendar month.

Precision/reliability needs

  • I will probably be fine with limiter being not super-precise, so it’s fine if limiting mechanism e.g. allows sometimes for 1050 per month and if it’s per 30 days from some arbitrary day, not exactly within a calendar month
  • On the other hand I won’t like if somebody uses monthly quota in one minute, better to leave some calls for the tomorrow visitors though good to allow small bursts.

So the ideal limiter would simultaneously apply something like:

  • no more than 1000 calls within 30 days or within a calendar month
  • no more than 100 calls a day
  • no more than 50 calls an hour
  • no more than 10 calls a minute
  • store whatever it needs in Postgres (okay to store it in-memory for a while, but syncs to DB are a must)
    • it’s a hobby project under active development, I don’t want to figure e.g. how to install and use Redis on gigalixir or fly.io while Postgres I am already using anyway
    • I don’t want to store state in-memory only - service is very likely to be restarted often. Actually in case of in-memory state even one restart mid-month could by accident overrun quota 2 times

What are the options?

I figured several options for proceeding and would love to get some advice from the more experienced folks.

  1. Hammer library (with a bit of massaging for combining the buckets) seems to allow for the rate limiting rules I am into. It doesn’t have a Postgres backend option, but maybe one is easy to write
  2. Create an own rate limiter/counter saving the log of requests (or the final counter) starting from mix phx.gen.context so the context would query call log table to tell the client how many calls are still allowed within month/day/hour?
    1. And if I’ll want to be “precise” over “fast” then maybe wrap it into a GenServer guaranteed to be run in 1 copy only
  3. Locate some other library that does approximately what I want. Maybe there’s something else besides Hammer that I just failed to locate so far.
  4. Something completely different?

What would you do in this kind of a situation?

Most Liked

gregvaughn

gregvaughn

I think you’re overthinking this. Write code that does what you described. I’d start with a database table that stores some token/label that represents a “bucket” of limits and a timestamp. Every time an API call is made you write an entry to that table. Before making the call you query for the limits – group by the various time intervals with a count of each and a having clause to enforce your desired constraints. It won’t be a trivial query (possibly window function or lateral join) but you’ll learn good things figuring that out. In the end it’ll be a single query.

artem

artem

Thank you, guys. I was also considering just writing a log of requests to the database and using SQL queries to check whether bucket is full or not. My main concern (that I should have stated more clearly in the beginning) was:

  • Am I reinventing the wheel? There could already be some known library to do just what I want or something very similar. Like if that Hammer library had a Postgres backend functional out of the box, it could be cheaper and more proper to just tune it to my case.

Now after this discussion it seems like my problem either isn’t common enough to have some known solutions or so simple that nobody bothered to make a library for it.

I guess most of hobbyists are using either totally free tiers or ones that have a clear spending limit. One I am likely to use is “metered”, so I could spend too much by accident. Probably it is not very common.

Four2

Four2

what would be easy to set up is a db chart of location, current temperature, date & time, clever remarks id
Then when the user lands on the page, a background query of that location’s date&time compare, then if it is within a certain amount of time since the last query of the temperature it displays the data you want on that chart line, if not, query the parse rss feed and update the temperature and the stored comments for the current weather, which would work well on a separate db table (like a remarks table with a unique column id) from the weather db table, a compare db table can be generated per location over time, and of course the remarks from another db chart.

Parsing data from a free xml/rss feed might be a better way of doing above instead of a web API.

zpeters

zpeters

I would highly encourage you to write this as its own package. This will be a great way to keep your concerns separated. I also think there are probably many others folks that could use a utility exactly like this!

Four2

Four2

Its not my project. But I would think getting the raw data from a free xml/rss feed would probably be better than someone else’s web API that you don’t have control over it and might change over time.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
wernerlaude
In AR this is so simple @articles = current_user.articles How to do in Ecto? def index(conn, _params) do current_user = conn.assig...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement