josevalim

josevalim

Creator of Elixir

Proposing Registry

Hello everyone,

I would like to propose the addition of the Registry project to Elixir:

The Registry project is a local and scalable key-value process storage in Elixir. It encapsulates 3 known use cases:

  • Process registry: to register process with dynamic names. Often Elixir developers need to rely on gproc or other tools.
  • Code dispatching: dispatch a module/function associated to a given key
  • PubSub implementation: send messages to local processes registered under a given topic

There are probably other use cases waiting to be discovered. :slight_smile:
You can learn more in the documentation:

http://elixir-lang.org/docs/registry/

The project clocks only 700LOC with documentation and performs well. We have extracted, improved and generalized the patterns from Phoenix.PubSub, the implementation used to manage and publish messages to 2 million subscribers.

When benchmarking thousands of processes registering serially, it is twice slower than local atoms, albeit 33% faster than gproc. On concurrent cases, it distributes well across all cores, becoming only 15% slower than local atoms, and 3x faster than gproc on a machine with two cores (gproc seems to be serial with its default configurations so the difference will be even bigger on more cores).

Please give it a try and let us know what you think.

Most Liked

josevalim

josevalim

Creator of Elixir

The Registry has been merged into Elixir master.

josevalim

josevalim

Creator of Elixir

This is local, the OTP one is distributed. You shouldn’t use the OTP one to store local data, as looking up or storing information may require messages across nodes.

On my benchmarks, the use of partitions have only been justified for the pubsub/dispatch use cases. For the unique registry, I couldn’t find a case yet where increasing the number of partitions matter. However, my current machine has only 2 cores. Maybe a partitioned unique registry may matter on machines with 16+ cores.

:gproc is a great project, albeit I find its API confusing (YMMV). If you asked me about a registry in Elixir six months ago, the answer would likely be no. So what has changed?

  1. I confirmed with the OTP team they have no plans to tackle a local registry
  2. While working on a separate project wth @chrismccord, we realized we could generalize the PubSub implementation while keeping the scalability aspects of Phoenix.PubSub
  3. The generalization of PubSub made it useful for at least 2 other common cases: :via lookups and module/function dispatching. :via lookups is a frequently required feature. Plus I am using module/function dispatching to replace GenEvent in another app and we will explore it instead of GenEvent in Logger too.

Being faster than gproc is a perk, although the registry was designed to scale with multiple cores. Work in the registry is always done on the client, there is no centralized entity, and that’s another useful pattern to have in hand.

In other words, there isn’t a single reason. We found a generic solution that is performant and scalable and our reaction is that it can be very useful as part of the language.

josevalim

josevalim

Creator of Elixir

Its API is quite simpler than gproc’s. There is no support for distribution as well, it is by definition local. One reason why we are hesitant on tackling any distributed registry as part of Elixir is because such is already part of the OTP team plans.

josevalim

josevalim

Creator of Elixir

@voltone and others have benchmarked the registry. I have consolidated @voltone results here:

Summary: on a machine with 40 cores, a Registry with 40 partitions provides across the board better results on concurrent registration. Without partitioning, the Registry starts to scale poorly when the concurrency factor is somewhere between 8 and 20.

Surprisingly, gproc performs quite well, even in concurrent scenarios. <speculation>I am assuming serializing writes ensures there is no contention for concurrent threads, so less coordination and process switches.</speculation>.

Erlang’s built-in atom registration does not perform well in highly concurrent scenarios but that’s fine. It was not meant to support dynamic names registration anyway (and doing so would certainly be a bug in your app!).

josevalim

josevalim

Creator of Elixir

Registry has been updated to v0.3.0 with:

  1. Improved performance for 1 partition (most common case)
  2. Allow a process to update its own value
  3. Support for named listeners
  4. Allow metadata to be stored along-side the registry

The last two features should make the registry more extensible. Listeners allows you to give a named process when the registry is started that will receive events whenever a process registers or unregisters a key. You could use those features to add Await support to the registry or even use the registry as a pool. We have added an example of using the registry as a sojourn pool to the examples directory. The pool is an example, do not use it in production:

https://github.com/elixir-lang/registry/blob/master/examples/sojourn.exs

The pool above measures the sojourn time, which is how long messages stay in each pooled process queue. That provides an idea of how fast processes are responding. Whenever there is a dispatch, we pick two random pooled processes and choose the one with the most recent reply and smaller sojourn time.

The pool itself works by starting a worker and a registry. The worker is a listener of the registry and a supervisor guarantees that both are restarted in case any of crashes. Whenever a new process is added to the registry, the worker is notified and starts sending sampling messages to that process which updates its own entry in case of crashes.

I hope it serves as inspiration for playing with the registry for other use cases.

Where Next?

Popular in News Top

whatyouhide
Hello folks, the Elixir team (well me, but I got a thumbs up from the others :P) is looking for contributors and potentially maintainer...
New
josevalim
Hello everyone, A new release candidate for Ecto is out. It fixes two regressions on the previous release candidate and adds three new ...
New
Elixir
1. Enhancements Elixir [Code] Emit :defmodule tracing event on module definition Mix [Mix] Add Mix.install_project_dir/0 [Mix] Add env...
New
Elixir
Code snippets in diagnostics Elixir v1.15 introduced a new compiler diagnostic format and the ability to print multiple error diagnostics...
New
Elixir
1. Enhancements Elixir [Code] Add :emit_warnings for Code.string_to_quoted/2 [File] Add :offset option to File.stream!/2 [Kernel] Auto i...
New
Elixir
Release: https://github.com/elixir-lang/elixir/releases/tag/v1.10.4 1. Bug fixes Elixir [Kernel] Fix a bug where custom types were prin...
New
Elixir
Release: Release v1.13.0-rc.1 · elixir-lang/elixir · GitHub 1. Enhancements Mix [Mix] Add Mix.installed?/0 2. Bug fixes Elixir [Appli...
New
Elixir
Type system improvements Type checking of protocol dispatch and implementations This release also adds type checking when dispatching and...
New
josevalim
Hi everyone, We are glad to announce that the first release candidate for Elixir v1.6.0 is out. Check out the CHANGELOG and give the ...
New
josevalim
Hi! :wave: I hope everyone is well! The Elixir team releases new versions every 6 months, typically every January and July. However, si...
New

Other popular topics Top

9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement