emoragaf

emoragaf

Best way to generate sequential ids

Hello, we have this situation right now:

Every time we make a sale we need to emit a document with a unique and sequential document number.

These numbers belong to a sequence based on the product sold (each document is for one product), we have a reserved range of numbers for each product, the ranges are big enough that we do not need to worry about running out of numbers right now.

We are looking at some options to generate these numbers (postgres sequences, Kafka events, Zookeeper sequence nodes), and I thought that maybe this is a good fit for Elixir.

I’m thinking of spawning a GenServer for each product configured with the sequence range and current value, so I can use it to generate the subsequent numbers.

I would persist the updated values to a DB to restore the processes in case of failure, and in case of needing some redundancy I could use something like libcluster + swarm to spread my GenServers across a cluster

Does this look right? Maybe I’m missing an obvious alternative, any pitfalls I might not be taking into account?

Most Liked

lalo2302

lalo2302

In my company we had the exact problem you have and we solved it using Postgres advisory locks:
https://www.postgresql.org/docs/9.4/explicit-locking.html Point 13.3.5

So you don’t deal with the concurrency on your elixir application, but on the database level.

So let’s say you want to have a sequential number for product “A”, and 2 concurrent requests come

Request 1 and 2 concurrently:

    1. Comes in
    1. Comes in
    1. Sets advisory lock with id :erlang.crc32("A")
    1. Sets advisory lock with id :erlang.crc32("A")
    1. Lock is being used, I’ll wait until it is free
    1. Query count “A” => returns 2
    1. Inserts new record with 3
    1. Finishes transaction and releases lock
    1. Lock is available, I can proceed
    1. Query count “A” => returns 3
    1. Inserts new record with 4

I hope I was explicit enough. Using postgres takes away you a lot of headaches. You don’t need to worry about if your new GenServer is a bottle neck, or if you deploy on a cluster how should you manage your processes.

shanesveller

shanesveller

Do the counters / sequences need to be durable across BEAM restarts, or internally consistent across multiple BEAM nodes? Both of those are challenging with any of the ideas above. Some other questions that come to mind:

Do they need to be monotonic across all machines?
Are gaps permissible?
Are collisions disastrous?
Can you overflow a given product’s range?

Most of these point me towards solutions on the persistent storage rather than application code, i.e. a specialized use of Postgres sequences or similar.

I’d also be curious what real-world or business constraints are at play here since the described technical limitations are unusual.

kokolegorille

kokolegorille

ksuid generate unique, sortable id.

You could (post/)prepend with your product id if You want.

Now your solution with a Genserver per product could work… it depends how much do You sell, but if You sell too much You might have a bottleneck. It will not work that well in a distributed mode.

It depends also if You just want an increasing counter, or uuid style id.

kokolegorille

kokolegorille

I would probably have a look at erlang counters if I had to do the same task.

emoragaf

emoragaf

The counters should be durable and consistent.

Having gaps is permissible but not ideal

Collisions are a big problem

Overflowing a range should not be an issue (range >>> sale volume)

My first thought was using something like Postgres sequences, but then I thought that having it on application could be a good fit and also help bring some devs in the team into the elixir world

Business constraints, in short we sell insurance policies, we have ranges of reserved policy numbers for different products from different companies, so we are limited to those IDs for public facing documents

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement