joshp

joshp

Correct way to structure PubSub broadcast data from a library

Hey all!

I’m building a library which will be listening to UDP events broadcast on the network, doing some parsing on them, and emitting the parsed events as Phoenix.PubSub broadcasts. The underlying UDP data has a number of different event types. My goal is to make it easy to handle all events emitted, or a specific type.
My question is about what the correct way to structure the pubsub broadcast to accommodate this is.

The typical PubSub structure is

PubSub.broadcast(:my_pubsub, "user:123", {:user_update, %{id: 123, name: "Shane"}})

My first pass was to do something like

PubSub.broadcast(:my_pubsub, "libraryname:udp", {:event_type_name, %{id: 123, data: "goes here"}})

But the problem is that when I subscribe to the “libraryname:udp” channel, I then need to pattern match for every single {:event_type_name, event} that might be broadcast, and have no way to easily pattern match for any event. (No way to differentiate an {:event_type_name, _} broadcast by my library from an {:event_type_name, _} broadcast by something else, unless I inject the topic name into the payload and match against that too, which seems inelegant.)

So my next attempt was to mimic the %Phoenix.Socket.Broadcast{} struct, and do this:

obj = %{topic: "libraryname:udp", event: :event_type_name, payload: %{id: 123, data: "goes here"}
PubSub.broadcast(:my_pubsub, "libraryname:udp", obj)

which allows users of the library to easily pattern match for all messages with:

def handle_info(%{topic: "libraryname:udp", event: event_type, payload: payload}, socket) do
    Logger.debug("Got event type #{event_type}, with data: #{payload}")
    {:noreply, socket}
end

or further match for specific event types by matching against the event type as well as the topic.

Now for my case, as a user of the library I’m writing, this is preferable! If I weren’t publishing the library I’d just go with it and not care. However I’m curious if this would be considered poor form for a library.

If the only thing that matters is documenting your output, then great, but I have the feeling I might be violating a norm by emitting events over a Phoenix.PubSub in an unexpected format, and I’d prefer not to have to re-write the library later with a breaking change just to do it the Right Way™. :slight_smile:

Marked As Solved

lud

lud

what if multiple processes need to consume the data

What if one consumer is a java app athat needs the even in kafka?

And what if you do not want a bottleneck gen server for the latest data but an ETS table ?

You can never cover all the cases I guess, so the best is to not close doors.

If you really want your library to force usage of Phoenix.PubSub then maybe accepting an option for the topic is ok. Or a topic prefix maybe. And then a tuple {YourLibrary, :actual_name} as the event name could work, so if the user provides a topic that other publishers publish to, there is still a way to match your library’s messages.

Edit; but I would just do both. Accept a callback where the default implementation (if no option given) is a broadcast to Phoenix.PubSub. And pass all the options given to your library when calling the callback.

Also Liked

lud

lud

For a library I think that the best would not to publish to phoenix pubsub at all.

I guess your UDP listener is running is a process, for instance a GenServer using gen_udp. In that case, your start_link/child_spec function should accept an option called event_handler or something like that, and the user can give their own implementation.

There are different ways to do that. The simplest is when a user will give a fun, and you just call the fun with an event like %{event: :some_event, payload: %{id: 123, data: "goes here"}}. Then the user can filter the events if needed, and dispatch to their own pubsub or do whatever they want with it.

You can structure your event data as you want because there is no more norm to violate in this case.

Another possible way is a callback module. You define a behaviour and the user implements that behaviour with their own module and give the module as the option value. So you can define different callbacks like handle_event, handle_closed. This allows the behaviour to define an ìnit function so the user can return an initial state that you would pass to further callbacks and update when the callbacks also return a state (like in a GenServer handle_call when you return a new state). This may not be needed at all. Also a fun/2 can do the job, when one argument is the callback type.

Finally, depending on your architecture, the user-defined handler can be a process. Your option can accept a pid but also an atom or a via-tuple so if the user process crashes and is restarted with a new pid but the same name, you can reach it. With a process, you just send your events as messages to that user process.

I’d start with a fun though. It’s simple and goes a long way.

joshp

joshp

Yeah, after thinking about it a bit you make a great point. Locking in a specific way of doing things make it a much less viable library.

Also, doing both and using the tuple event name seems like a pretty perfect solution for all use-cases, I’ll give it a go.

Thanks for the help, really appreciate the ideas!

Where Next?

Popular in Questions Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement