Cruz

Cruz

How to avoid the performance impact of large ETS-based queries?

Hi,

I’m writing a service in which each transaction starts with the system returning a “catalog” of available products to each caller/client. There are more than 1K products and 11 different of these catalogs. Each product contains a series of IDs, name and descriptions in two languages, multiple flags, and other data. In other words, the size of each catalog is considerable.

Given the frequency in which the catalogs are used, I thought to load them into ETS tables for quick access. However, I found the following article:

https://medium.com/@jacob.lerche/using-constant-pools-to-speed-up-your-elixir-code-c527d533c941

The author suggests using a macro when the size of the data is too large for ETS. However, he doesn’t provide an idea as to how much is too much data. And, using macros implies having to recompile every time the data changes. This might be OK, but I prefer to avoid it if possible.

Does anyone has found this issue before? Do you know when the size of the data becomes an issue for ETS?

I plan to do some load testing with the ETS based solution. Any other suggestion?

Thank you

Marked As Solved

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Hey @Cruz. Does each transaction load the entire catalog or just specific items within it? Specifically, the article is talking about how the copying penalty happens when the items you want to look up are themselves very large, not necessarily when the whole table is large. If the whole ets table is large but each item is small, and you only want a few items, those are the only items that are copied.

These days however if it’s a very constant sort of thing I’d look at using http://erlang.org/doc/man/persistent_term.html You get basically the same performance and copying characteristics without having to deal with macros.

Also Liked

jacoblerche

jacoblerche

Hey there, author of the article here. Ben already gave very comprehensive answers, I’ll just add a few points as to when to use constant pools

  1. Large data that needs to be read by a lot of processes. If it’s just a handful of processes, you might be better off with ETS or something else, unless the data is gigantically large
  2. Data that still needs to be updated regularly, but infrequently compared to the reads

I should note that compilation of a module with static data is actually deceptively fast especially if you use Module.create/3.

I should also point out that binary data benefits from reference counting. IIRC, if the binary is 64 bytes or greater only a reference to it is copied over to a process. Just another thing to possibly consider if it fits your needs.

michalmuskala

michalmuskala

I saw the mention of :persistent_term, so I’d like to underline that dynamically compiling modules with static data now that we have :persistent_term is in almost all of the scenarios going to be slower and will do more operations than using :persistent_term. I’d consider that technique to be largely obsolete now.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

@Cruz There’s a final option here which is to treat the catalog like a file and put it on a CDN with appropriate cache headers. If it changes rarely and clients are of the type that can cache values locally then in an optimal world they shouldn’t even be reretrieving it all that often.

wmnnd

wmnnd

@evadne did a presentation at this year’s ElixirConfEU where she benchmarked ETS.
The video isn’t up yet, but here are her slides and the demo repository:

idi527

idi527

I plan to do some load testing with the ETS based solution. Any other suggestion?

That’s probably the best approach …

Also providing a sample of your data and the planned access patterns would be useful in case you run into performance problems.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement