fireproofsocks

fireproofsocks

Does Image Rendering "shell out"?

I’m hoping an Erlang/Elixir Jedi can help show me the ways of the force here. I got a bunch of images that need to be manipulated programmatically (stop me if you’ve heard this before). So I’m looking over the approaches written about here:
https://sergiotapia.me/generate-images-with-name-initials-using-elixir-and-imagemagick-374eca4d14ff
and here Elixir Native Interoperability – Ports vs. NIFs

Basically, the question comes down to this: is the image manipulation process “booting up” each time an image is sent to it for rendering? Or is there a way to boot up the image manipulating code (whatever it is) ONCE when the Elixir app starts, and then run each image through it?

In other languages, any time we “shell out” to an ImageMagick or similar command, the Image processing utility starts up each time the call is made. I’m imagining this like starting up Photoshop on my desktop: you launch the app, you see the banner image, the spinning wheel, the app loads, then the image editing occurs, and then it shuts down. Wash, rinse, repeat. However, we are crunching through millions of images, so that boot-up time ends up being prohibitively expensive (both in terms of rendering time and CPU).

Can someone help explain the differences between the different approaches available to Elixir? System.cmd() seems to represent “shelling out” – i.e. whatever it is calling, that executable needs to load up each time the command is called. Whereas NIFs and ports seem to be able to boot up once and stay running alongside the parent Elixir application. Is that accurate? And there might be some Elixir module we can use? Given the need for this process to be as performant as possible (and not reload executables redundantly or blow up our EC2 bill), what is the best way to accomplish a task like this?

(FYI: the image transformations are relatively simple: mostly flattening layers or doing simple resizing). Others have expressed concern about using Erlang for computationally-heavy operations, but I would like to at least give it a good honest benchmarking. If not Elixir/Erlang, C or Go or Rust (or ???) might be preferable solutions for handling high concurrency and being able to natively perform the image manipulations without “shelling out”.

Thoughts or suggestions welcome. Thanks!

Most Liked

NobbZ

NobbZ

I only glanced over the blog post, but yes, it is “shelling out”.

But to my experience so far, IMs boot up time is neglible.

If you really want to, you can of course try to interface the C++ library in a NIF or C-Port, but I am not sure if this will make a huge difference…

dimitarvp

dimitarvp

What you can do is always have 1 or more executables started via System.cmd and waiting on their stdio for data. That’s a trick I used in other programming languages when I was concerned about the initial overhead of loading a native executable in a child shell; because now the executable is loaded and is only waiting for data.

One example coming to mind was re-coding videos through ffmpeg; I’d start it with all necessary flags but without any input and it would just wait on stdio until I send it a video to process. Once sent, it will process it immediately.

Can’t give you exact details but that’s usually how a pool of disposable native processes is managed.

Where Next?

Popular in Questions Top

vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New

Other popular topics Top

peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New

We're in Beta

About us Mission Statement