elie

elie

Why use ETS? When not to use it?

My first question is what does ETS add on top of just storing state in a GenServer?

In the documentation I see it is used as a “cache”, does this mean it won’t always return the latest data?

There’s a warning in the Elixir guide about using it prematurely:

Warning! Don’t use ETS as a cache prematurely! Log and analyze your application performance and identify which parts are bottlenecks, so you know whether you should cache, and what you should cache. This chapter is merely an example of how ETS can be used, once you’ve determined the need.

I don’t have any performance issues at this time? Does this mean ETS should not be used? What are the downsides of using ETS?

Most Liked

sasajuric

sasajuric

Author of Elixir In Action

You could simulate the complete interface of the :ets module with GenServer. However, such implementation would be very inefficient in some cases.

A typical example is in memory k-v. If you do this with a single process (GenServer or Agent), then all access is serialized. If you have thousands of client processes interacting with such k-v, operations are performed one at a time.

In contrast, an ETS powered table with proper knobs turned on (:public, :read_concurrency, :write_concurrency) will allow different clients to interact with the same table simultaneously. Multiple processes can issue reads and writes to the same table at the same time, and the operations can be performed simultaneously, unless both processes are writing to the same row.

An example of this in practice is Registry, which is powered by ETS tables. This ensures that client processes can quickly find desired processes, and even get some of their properties, without needing to message some process.

The thing @rvirding and @Nilithus mentioned about avoiding large process heap is also an interesting feature. ETS data is “off-heap”, so it doesn’t put any pressure on GC. If you have a single process with a large, and frequently changing, active memory set, ETS table might improve your performance significantly.

So basically, ETS is mostly an optimization technique. If your needs are simple, you can probably start with a GenServer, and consider ETS if you find performance problems.

26
Post #8
rvirding

rvirding

Creator of Erlang

Using ETS as a cache is only one of its many uses. A major use is for storing LARGE amounts of data in memory. This is a typical use, for example it is how mnesia stores data in memory. So if your gen_server stores large amounts of data then maybe using ETS is a better alternative.

As has already been pointed out ETS tables can be shared between processes. One thing to watch out is that ETS is a datastore not a database and the support for transactions is VERY limited. If you need to implement transactions you will probably need a process in front of the ETS tables.

There are many other uses of ETS. Another one for example is having supervisor use ETS tables to share data between its children, both current and future ones.

So in Erlang ETS is used quite often.

23
Post #4
Nilithus

Nilithus

No expert here – but from the reading I’ve done 90% of the time you’re better off just storing state in the GenServer.

There is a couple of interesting use cases since ETS tables have some access control:

public — Read/Write available to all processes.
protected — Read available to all processes. Only writable by owner process. This is the default.
private — Read/Write limited to owner process.

so you can have shared state across processes. I believe the Elixir Registry uses ETS tables under the hood.

The other use case I’ve seen is if you are highly concerned about Garbage Collection(GC) cycles the ETS table is not stored in the process’s heap. This Erlangelist blog article talks about saving GC latency when you have large data buffered in a process.

But these are all pretty big edge cases I think. Hope this helps!

10
Post #2
minhajuddin

minhajuddin

You can use ETS in many places where you use your database depending on your use cases. One easy win is putting all your lookup data into ETS tables. ETS is very fast when you do key lookups. I do a little bit of ETL and use ETS tables extensively to store lookup data, to write out to ets tables as an intermediate store etc,. I also use them in an image_downloader app to keep a list of URLs which have been downloaded with their checksums and headers so that I don’t need to download them the next time I see them. I use :ets.file2tab('./data/myetsdb.ets') as soon as I start the app and do a :ets.tab2file(:mydb, './data/myetsdb.ets') before I shut down or run into an error.

minhajuddin

minhajuddin

You are right (I am not sure what info about the modules is stored). There is a lot of information stored in ets tables. Simply opening iex will create ~19 ets tables.

iex(1)> :ets.all
[Logger.Config, IEx.Config, :elixir_modules, :elixir_config, :file_io_servers,
 :inet_hosts_file_byaddr, :inet_hosts_file_byname, :inet_hosts_byaddr,
 :inet_hosts_byname, :inet_cache, :inet_db, :global_pid_ids, :global_pid_names,
 :global_names_ext, :global_names, :global_locks, 4098, 1, :ac_tab]
iex(2)> :ets.all |> Enum.count
19
iex(1)> :ets.tab2list :elixir_modules
[]
iex(2)> :ets.tab2list :elixir_config 
[{:home, "/home/minhajuddin"}, {:erl_compiler_options, []},
 {:compiler_options,
  %{debug_info: true, docs: true, ignore_module_conflict: false,
    relative_paths: true, warnings_as_errors: false}}, {{:uri, "http"}, 80},
 {{:uri, "tftp"}, 69}, {:argv, []}, {:at_exit, []}, {{:uri, "sftp"}, 22},
 {{:uri, "ftp"}, 21}, {{:uri, "https"}, 443}, {{:uri, "ldap"}, 389}]

Where Next?

Popular in Questions Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New

We're in Beta

About us Mission Statement