OrontaMedu

OrontaMedu

Long running tasks and external processes: in Elixir or Rust?

In a project there’ll be lots of background tasks run by DynamicSupervisor.

Each task will be making network requests to an external service and is expected to run for a few or 10-20 minutes. And a task will need to save a result into a DB.

The way a task makes an network request could be in one of 2 ways:
(1) via an Elixir library – ExternalLib.make_request(a, b, c)
(2) via an non-Elixir library (in Rust): System.cmd("external_utility", [a, b, c])

Note that both (1) and (2) are black-boxes – they may not be modified. They may only be used.

The (2) in this case is more flexible, faster, more properly implemented and generally is preferable – compared to (1).

Question: from the point of view of Elixir, background tasks, ability to run, manage them, get a return result and so on … – could the Elixir library (1) be better still than the non-Elixir one (2)?

Most Liked

derek-zhou

derek-zhou

Much have been said about you should stick with a pure Elixir approach, then only move on to Rust NIF when there is significant benefit in doing so. I want to point out the CGI style System.cmd interface is most likely not what you want, should you decide that neither pure Elixir nor Rust NIF is good enough for you.

If you must use an external process, then the only valid reason is that you need to keep significant state in that external process. The best way to communicate with a stateful external process is via a port. Rust offers an excellent async library, Tokio, to make that happen. You would need to write a couple hundred lines of Rust code to wrap the Rust library though.

D4no0

D4no0

Are you willing to sacrifice all the benefits of elixir for some performance? If yes then go with 2.

LostKobrakai

LostKobrakai

Yes. With external programs you have to marshall communication in both directions. Staying with elixir means you don’t need to do that.

dimitarvp

dimitarvp

Not sure you can have something be faster when it hits the network. The CPU is just waiting for a response.

And in your case you don’t even have a Rust library to call in-process, you have an external tool.

I’d say stick to Elixir unless the Rust tool is bringing critically important benefits.

dimitarvp

dimitarvp

I am prone to being preachy and absolutist sometimes so maybe I am not the best person to give an answer… but I am leaning to “yes”.

To me calling something in-process has less things that can go wrong. Though seriously, tools that are well-behaved are probably just as safe. (“Well-behaved” in this instance means “they respect shell conventions when having to stream data in/out from/to stdin/stdout”).

Of course I’d also seriously consider just doing the whole thing in Elixir if I can get away with it. As mentioned above, disk or network I/O are 1000x more time-expensive compared to CPU-intensive tasks so who cares if your code that will need 350ms to get back an answer from an external server is written in Rust, Go, Haskell or even Ada or Brainfuck?

Even more supporting my conclusion then. I/O-bound tasks can be written in whatever. You don’t need Rust for that; Rust is for when you need every last bit of performance and minimal memory pressure.

Where Next?

Popular in Questions Top

logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
Kagamiiiii
Student & New to elixir. Nice language. I want to convert a english character, e.g. “a”, which is stored in a variable, to it’s asci...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
wernerlaude
In AR this is so simple @articles = current_user.articles How to do in Ecto? def index(conn, _params) do current_user = conn.assig...
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

We're in Beta

About us Mission Statement