alexcastano
Is a singleton ETS a good idea?
Using libraries like singleton you can create applications which are the only one in the whole cluster. This application may be a GenServer which initializes an ETS table and delegates the requests to this ETS. It is like a thin wrapper to convert a local ETS into a global one.
I have this doubt because I read the following in the hammer documentation:
There may come a time when ETS just doesn’t cut it, for example if we end up load-balancing across many nodes and want to keep our rate-limiter state in one central store. Redis is ideal for this use-case, and fortunately Hammer supports a Redis backend.
If you need a shared cache for an elixir cluster, what do you prefer to use a Redis server or a singleton ETS or even a singleton cachex?
I can think advantages and disadvantages of both methods.
- With singleton ETS or Cachex you can save Elixir structs or any erlang term, in redis you have to
castdata from database to memory, unless you use very basic data structures like strings or numbers. - Singleton ETS won’t be as performant as a local one, but I think in most of the case could be enough. Maybe is it faster than Redis anyway?
- With Redis you have to create a bigger wrapper to use it like cache.
- You lose your cache if the node dies. This can be acceptable in some scenarios.
- Less architectural dependencies.
- Redis has built-in some interesting features like TTL.
I didn’t test it, because of that my question is: Is a singleton ETS a good idea or it has his own drawbacks? Or would you prefer Redis?
Most Liked
cmkarlsson
The singleton linked is just a wrapper around the global module. It has its own set of problems, mostly that it can’t handle split brain scenarios and it is comparatively slow.
I don’t think you should write off mnesia based on that, it is not written to discourage but to show some things you may need to handle depending on use case. It is a great solution for distributed caching and the reasons behind these points is that there are well known “quirks” you may want to be aware of. Better the devil you know as they say. Most other distributed setups will have some set of problems but you may not be aware of them. If you go down the global over ets you may end up re-inventing mnesia.
This externalizes the problem. Instead of dealing with net-splits, high availability and performance in your BEAM cluster you needs an OPs team handling in externally. You may need to setup multiple redis nodes and either have a client being able to swap over to another node in case the master node goes down or setup a virtual IP to handle it which requires more infrastructure.
The question is why you need the distributed cache. Normally it is either because of performance or high availability. If it is for performance then anything cached on the local node will beat redis. If it is for high availability then it is either internal or external complexity.
If you don’t require the HA setup for redis you can just run a cache on a single node (with mnesia for example) and connect directly to the node-name. Then you have a similar setup as a single instance of redis.
alexcastano
I found this benchmark which answers my own question, very interesting:
pggalaviz
Sorry for the late reply.
I think what @jordiee tried to say is that in the benckmark, all ETS calls are going through the GenServer’s callbacks, effectively generating a bottleneck.
If instead of calling the GenServer you just call the ETS directly (eg. RPC call to the node who owns the ETS) you’ll get better performance, the GenServer will only be used to start and own the ETS.
I wrote a blog post about a toy distributed rate limiter in Elixir that better explains it: http://pggalaviz.com/2018/11/26/simple-distributed-rate-limiter-with-elixir/
alexcastano
Agree.
This has other problems. It is known that cache is a complex matter, if each node has its own cache could be even more complex:
- Imagine a rate limiter like
Hammer.Plug, you need the data to be centralized. - Or if you have “bearer token” sessions in this cache and an user deletes one. You should delete in every cache. (You can have an intelligent load balancer to distribute the request by user but this is OP complexity and it is not always possible)
HA is not my case.
That’s actually a very good idea. I’ll take a look. Anyway, mnesia seems more complex than ETS for a key value store.
My point is distributed cache is a complicated matter and there are many situations where a single cache for the whole cluster it is enough and much simpler.
alexcastano
Yeah, you are absolutely right 
I thought it was weird… 
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"