almir-neto

almir-neto

There is some advantage to use `File.open` instead `File.write`?

I know that File.write performs two operations: it opens the file and then writes to it. However, considering that File.open followed by IO.write avoids one operation for each iteration, I want to know: when I’m building a lazy pipeline, is it recommended to use File.write for each iteration, or should I keep the file open and just write to it? Is there any significant impact between these two approaches?

Could someone clarify the pros and cons of these two approaches for me?

PS: I am focused on impacts for lazy pipelines like this:
Stream.map(…)
|> Stream.reduce(…)
|> Stream.with_index(…)
|> Stream.map(write_on_file)

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

In these sorts of patterns I would also checkout Stream.into(File.stream!(path))

dimitarvp

dimitarvp

I’ve used File.open and IO.write / IO.binwrite with a Stream like you, dozens of times, and never benchmarked it against having File.write at each step. I simply assumed the latter is a terrible idea and can never be faster. Main reason might not even be speed per se; it’s just that opening and closing a file can be risky under load and in more constrained machines (which, let’s be real, are most cloud offerings out there, even the mainstream ones; they regularly put your apps and databases on I/O-throttled machines) – it might actually fail with “out of memory” or “not enough file handles” etc., I’ve seen those even on machines with 64GB of RAM and 2TB SSD storage.

TL;DR do as little syscalls as at all possible. We can’t optimize adequately for that when we work with a higher-level language with Elixir but at least we can use the little tools we have, like “don’t open and close file 500 times in a loop”.

…And all of that changes if each step to write to the file is seconds or minutes apart, of course (i.e. you stream data from a slower remote node).

lud

lud

I’m not sure I understand the question. If you need to do multiple writes to a file in a loop then yes, File.open is made for it. If you have the complete content of the file to write already, then use File.write only once.

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New

We're in Beta

About us Mission Statement