thinkingcat

thinkingcat

Offline-first sync on client -- library?

Hi, I am looking for a library for Elixir that helps to do offline-first type sync to the client. We are currently using Ecto and Absinthe but I don’t think subscriptions can help with asynchronous updates. We can do Listen/Notify, but I haven’t looked into that yet. Hoping there is someone out there who’s thought of this issue already and is working on it as there are a number of good alternatives in other languages :laughing:

We are running Flutter, and this is an example of a synchronization library that is available in Python:

Examples in other frameworks are Watermelon DB, Realm and Parse.

Research in Past forum posts have shown me something like Syncing deeply nested list of maps with client, which is very much ‘in the weeds’.

I’m looking for, as I am sure others might like, an easy plug and go library for syncing the database of a offline-first client to the server.

Thanks! :slight_smile:

Most Liked

mdwaud

mdwaud

The Martin Kleppmann suggestion is great - his book is excellent as well.

I work on a similar problem where we have distributed devices running Elixir and Nerves that have intermittent connectivity, but need to reliably transmit data to and from the cloud. The approach I ended up taking was very similar to Kafka or database replication, where data is written to a sequential log on either end and the log is asynchronously replicated between client and server with at least once delivery. This approach solves many problems (reliable delivery, easy to audit or reconstruct system state, etc.), but introduces new ones (linearizability, immutable data, write amplification, etc.). Building on top of this you can use CRDT type approaches like a PN Counter to make eventually consistent values. Beware, though, distributed systems can be a real pain in the neck.

The system runs in production quite well and there is an example app here, however I am sharing this mostly for educational purposes since the protocol is still changing I’m not sure how much support I can give.

It’s a really interesting problem, though, and I’m happy to let you bounce any ideas you have off of me.

ruslandoga

ruslandoga

Subscriptions, listen and notify all require network and wouldn’t help you when the user is offline. Offline-syncing is very much depends on complexity of your data structures, how conflicts are to be resolved, whether you want to only push offline writes from a single client per object (easy), or you need to be able to resolve conflicts between multiple clients to the same object (hard), etc. I’m not aware of any off-the-shelf solutions in elixir ecosystem. There are some CRDT and the recently released Delta which seems to be OT libs out there but any of them might be not suitable in depending on your usecase.

Off-topic – there are really good lectures on youtube by Martin Kleppmann on this and related topics.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New

We're in Beta

About us Mission Statement