fklement

fklement

Most efficient way to check for existing data entries

Hello!
I’m currently having a business case where I periodically (about every 5-10 seconds) will need to check if the data that gets inserted does exists in the database. It will also be mostly the case that indeed the data was inserted before.
Now I’m questioning myself what would be the most efficient way to handle that.

Here some solutions I got up with:

  1. *Using |> unique_constraint(:device_id) in the changeset

  2. *Setting an index on the unique identification criteria (every tuple to be inserted has it’s own device_id)

  3. *Trying to fetch data with the given identification criteria (device_id which is known in advance)

  4. *Keeping a state with all currently in the db existing device_ids and checking against that

Would be cool if someone can give an elaborate explanation.

Most Liked

rjk

rjk

As far as I understand the context/your use case,

I would definitely go with the unique_contraint and index on the table on the device_id column to have the most simple and correct way of lookups for these devices. (first make it work and correct)

To go faster (after making it work and correct) you could deploy various techniques from very simple to more complex (giving more headaches because of higher risk of race conditions)

Simplest in my mind would be a cache with a TTL(time to live) for every record you lookup, so you’ll always get a database hit on the first try and then it ‘sticks’ for x seconds or minutes within your cache. This is only applicable if it’s allowed to let a record / device access? linger after a deletion for max TTL timeout. If this is totally now allowed then you need some mechanism to update it after a deletion (See postgres pub/sub listen/notify below)

Now if you want to make it really fast, even on a first lookup, you could create a simple map (fast lookup) on the elixir side if you don’t expect too much of them (else it would take up too much ram), now to keep this up to date you could refresh it completely every x minutes (again, depends on how big the dataset is), otherwise you could use notify/listen (if you’re using postgres) to do a very lightweight pub/sub on modifications of your table.
(see for examples here and here)

To avoid data races, best if you always do your writes directly to the database and ignore your cache. The cache should only be updated by your pub/sub mechanism or full snapshot per x seconds. And you could always treat your cache as the true positive case, as in; if you can’t find them in your cache you do the ‘slow path’ of looking them up in the database. In this case you should always be correct and fast if it’s a returning client.

I’ve used the above technique myself to cache access tokens on the elixir side (which can refresh themselves out of the http request response loop) making the API calls in the microseconds because they don’t have to do a roundtrip to the database to check if they are still allowed. In this case it doesn’t matter if somebody’s access get revoked that it takes a couple of seconds before that cache updates itself and if it’s not found in the cache I always make a database call to see if it exists. So it’s really speeding up the case where API clients return often with a call.

Conclusion, it always depends on the context and requirements but always go for working & correct first, time it and see if it’s fast enough because the database will also keep a lot of your data hot in memory (last fetched records and your index most of the time) so it should be fast enough, if not you could always put in a lot more effort :slight_smile:

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New

Other popular topics Top

jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New

We're in Beta

About us Mission Statement