ijunaidfarooq

ijunaidfarooq

Writing 1K images to disk using Task.async_stream

I am downloading images through HTTP request with which I am getting a binary image, and writing it to a file such as

File.write(image_with_dir, image, [:binary]) |> File.close

this whole operation of getting HTTP request and then writing it to disk is done in

|> List.flatten()
|> Enum.sort()
|> Task.async_stream(&(inline_process.(&1, images_directory)), max_concurrency: System.schedulers_online() * 2, timeout: :infinity)
|> Stream.run

When decreasing max_concurrency the process got slow approx 2 minutes, also results of System.schedulers_online() is 8

but with current max_concurrency it faster but with this. Disk IO starts touching the limits

Purpose of writing those files is to send them to Dropbox with a batch of 1000 as dropbox upload session supports 1000 images at a time.

Is there any better way to write images to disk? maybe in memory but I don’t know, any help would be wonderful also this operation is being done on Cuda GPU machine but I am not sure how I can use GPU for such purpose.

This process is user defined. user can ask for less/more than 1000 images and those can be one or multiple Task.async_stream’s…

I want to save them on disk so if the process broke or application gets a restart, I can resume the download from where it left.

Most Liked

idi527

idi527

A file opened in raw mode wouldn’t work with IO.write and IO.binwrite, I think?

iex(1)> {:ok, fd} = File.open("some.file", [:raw, :write])
{:ok,
 {:file_descriptor, :prim_file,
  %{
    handle: #Reference<0.3732712904.755892237.167770>,
    owner: #PID<0.105.0>,
    r_ahead_size: 0,
    r_buffer: #Reference<0.3732712904.755892226.167656>
  }}}
iex(2)> IO.write(fd, "hello")
** (FunctionClauseError) no function clause matching in :io.request/2

    The following arguments were given to :io.request/2:

        # 1
        {:file_descriptor, :prim_file,
         %{
           handle: #Reference<0.3732712904.755892237.167770>,
           owner: #PID<0.105.0>,
           r_ahead_size: 0,
           r_buffer: #Reference<0.3732712904.755892226.167656>
         }}

        # 2
        {:put_chars, :unicode, "hello"}

    (stdlib 3.12.1) io.erl:565: :io.request/2
    (stdlib 3.12.1) io.erl:63: :io.o_request/3

:file.write can be used instead.

iex(2)> :file.write(fd, "hey")
:ok

Also there are some ownership complications similar to sockets where only the process that opened the file can write to it, not sure if it’s a problem for the Task.async_stream approach described in OP (it shouldn’t be a problem if the file is opened and worked with within the same task process).

axelson

axelson

Scenic Core Team

Since it sounds like you don’t need to persist the files in memory, you could mount a ramdisk so the files never have to touch the disk (assuming you have enough ram)

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement