tmock12

tmock12

Replacing Rails Background Jobs with an Elixir Service

Need opinions on something. One of our projects is a large Rails monolith with a ridiculous amount of background jobs. We keep creating additional jobs and adding features to them. More and more I find myself thinking “… if this was Elixir we could do this and that more efficiently…“. I’ve seen talks in the past about replacing parts of their Rails ecosystem with Elixir services.

Has anyone done this as a goal of replacing background jobs with Elixir processes? Any thoughts on that experience?

Doing a full Rails replacement would be wonderful, but is unfortunately not feasible at this point. Is introducing a new tech stack worth it? Would love any opinions from anyone that has introduced Elixir as a service to their existing apps.

Thanks!

Most Liked

hubertlepicki

hubertlepicki

Yes, that’s precisely why doing events / reactions versus background jobs is better approach IMHO. You don’t schedule a job, you send out an event somewhere. That somewhere is a queue. Then some things can pick up that event and act upon it. The major advantage of thinking this way is that you are detaching the event from implementation. If you save to database module name / function / arguments to be executed, you are also prone to the code changes - i.e. suddenly the module is not available after deploy, or it’s taking now 3 not parameters and your background queue is going to get messed up.

If you are writing your Rails app properly, i.e. there is not much logic in the models themselves, and you have stuff extracted to services, you can quite easily connect from Ecto to the same database Rails uses. It’ll understand it well, and can manipulate the same data Rails does. You obviously might need to re-implement some of the logic in Elixir but generally that’s what I would also do in Rails - background job services would be implementing own logic wherever possible to keep it isolated.

axelson

axelson

Scenic Core Team

In case anyone is still coming to this thread (based on views it appears so). Another good option for running persistent background jobs in Elixir is https://github.com/sorentwo/oban/ (this is the job library that I like to use).

anthonator

anthonator

If you’re wondering if you can do this purely in Elixir, you can. However, you are going to need to roll your sleeves up and get dirty learning OTP. Personally, I think implementing a job queue/event dispatcher is a great way to learn Elixir/OTP. The use case hits on most of the OTP sweet spots and requires you to dig a little.

You’ll need to put some effort in up front to understand how advanced you want your queue to be. Specifically, how durable should your jobs be and should they be distributed? If your server crashes or restarts is it OK to lose all your jobs? Do you need/want multiple servers handling jobs? If you need durability and distribution then you’re probably better off reaching for something like Exq or Honeydew. If not then Task’s might be a good starting point.

hubertlepicki

hubertlepicki

I/my team have done pretty much this twice, two different approaches. In first project we used RabbitMQ queues for background jobs. This was a design decision from start, so we could have workers processing those written in whatever we wanted. This is a good approach if you are already using RabbitMQ.

If you are not, but you are using Resque or Sidekiq, there might be easier path by using https://github.com/akira/exq - I only have limited experience with that library myself, but from what I have been told it works as advertised.

anthonator

anthonator

Reading back to your original question, are you running into specific problems with your Rails job implementations? You say you think you could do some things more efficiently in Elixir, but are there any actual problems on the Rails side? Is there a goal you’re trying to accomplish that moving your jobs to Elixir would help with (i.e. infrastructure costs)? Might be a premature optimization if you don’t see or aren’t expecting any issues.

Where Next?

Popular in Questions Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
wernerlaude
In AR this is so simple @articles = current_user.articles How to do in Ecto? def index(conn, _params) do current_user = conn.assig...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New

We're in Beta

About us Mission Statement