ConnorRigby

ConnorRigby

Nerves Core Team

Help implementing a notification system for Sqlite ecto adapter

So i’ve landed some features

In the lower level libs that make up the Sqlite.Ecto2 adapter. Now my problem is one of the actual implementation when working with Ecto. I know Ecto itself doesn’t support any notification system, but
the Postgrex adapter does support notifications. From skimming the source, i found that Postgrex just
opens another connection to the database which is something that sqlite can not support just because how it was built.

so one idea i had was to use Elixir.Registry, so every connection in the pool sends notifications out via Registry.

My issue is that sqlite sends a message that looks like
{:insert | :update | :delete, 'tablename', rowid}
(here are the docs for that)

Basically i get a rowid not a primary key or anything that is actually exposed to the Ecto Repo,
so even if an end user subscribed and got this message, they wouldn’t be able to easily lookup the data
that was affected because rowid is kind of an internal thing.

What i was thinking of doing is:

%{columns: c, rows: [row]} = Ecto.Adapters.SQL.query!(repo, "SELECT * FROM '#{table}' where rowid = #{rowid};", [])

And then sending that message to subscribers of the Registry, so a consumer would have to do:

receive do
   {action, 'users', %{columns: c, rows: [row]}} -> MyRepo.load(User, {result.columns, c, row})
end

The only problem i have with this is that every insert will also have a query to dispatch the notification,
and having user call load seems a little strange also.

Most Liked

ConnorRigby

ConnorRigby

Nerves Core Team

unfortunately the implementation of the message dispatching is handled by a NIF, and the convention for that NIF is to use charlists, because the lib is an Erlang library, not Elixir. Charlists are also faster for NIFs if i remember correctly, since it doesn’t have to allocate, convert to term, deallocate etc.

I don’t think Ecto encourages looking up a schema module at runtime, since multiple schemas could use the same table.

I guess dispatching the rowid isn’t that bad. I could make a wrapper module that one could use by supplying a schema and translate it for the user.

ConnorRigby

ConnorRigby

Nerves Core Team

I don’t think its a private api, just part of how the Sqlite3 virtual machine works. You can read about rowid here. Basically it exists, and you can use it to index things, but you can’t look at how it works. (as far as my limited knowledge of the sqlite public api goes anyway)

In most cases the rowid is the primary key, (although this isn’t always true), the only problem is that sqlite internally doesn’t actually dispatch the callback (what sends data back to Beam) until after the row has been destroyed.

There actually is commit and rollback hooks (described here) but that would add more complexity to the NIF than i would ideally give it.

idi527

idi527

That would be perfect, at least for me, I think.

ConnorRigby

ConnorRigby

Nerves Core Team

The nif handles messages in a thread, so it’s possible that messages can come in faster than they can be handled, but they should be received in order making this not that big of a deal i don’t think

dimitarvp

dimitarvp

Fair enough, I figured. At this point you’d basically be making Sqlite++ with half the code being the original Sqlite (C++) and the other half Elixir. Far from ideal.

So it’s kind of a private API? Maybe you can ask the maintainer if you can make use of that journalling metadata before it gets GC-ed?

Sorry I am not very helpful, mostly just thinking out loud.

Where Next?

Popular in Questions Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New

We're in Beta

About us Mission Statement