polypush135

polypush135

Streams for processing a tree of folder and file paths?

I’m looking to learn the File module and was trying to think of a pet project for doing so.

I have a huge dir full of mp3 files that I want to organize.

Given the nature of music Artist → Album → Track.
I figured I would be building a possibly big list of folders and files into memory.
So naturally the first thing that comes to mind is streams.

Is there something like File.ls but for streams?

Or maybe a better question. What kind of limits are there to the number of folders and files that can be made in any one folder on today’s OS systems?

How does File.ls handle large lists in other words?

Marked As Solved

al2o3cr

al2o3cr

I’m not aware of anything built-in, but there’s erlang-dirent:

You’d likely want to wrap this in a module that does a couple things:

  • translates the input from a binary to a charlist

  • uses Stream.resource etc to produce a Stream of matches

  • transforms the elements of the stream back into binaries from charlists

Also Liked

al2o3cr

al2o3cr

Here’s a standalone demo:

Mix.install([
  {:dirent, git: "https://github.com/team-telnyx/erlang-dirent.git", branch: "master"}
])

target_path = "."

Stream.resource(
  fn ->
    {:ok, dir_ref} =
      target_path
      |> String.to_charlist()
      |> :dirent.opendir()

    dir_ref
  end,
  fn dir_ref ->
    case :dirent.readdir_type(dir_ref) do
      :finished ->
        {:halt, dir_ref}

      {:error, reason} ->
        {[{:error, reason}], dir_ref}

      {name, type} ->
        {[{List.to_string(name), type}], dir_ref}
    end
  end,
  fn _ -> :ok end # not used because :dirent cleans up on GC
)
|> Stream.each(&IO.inspect/1)
|> Stream.run()

Followup thoughts in no particular order:

  • This uses :dirent.readdir_type since the first question that code that consumes the stream is likely to ask is “is this a directory?”

  • The error handling is somewhat inconsistent; a failure in opendir will crash but a failure when iterating through the results will put {:error, reason} in the output stream. Your application may have different needs.

  • the situation with filenames that can’t be represented in UTF8 is complicated so this code completely ignores it. That may not be sufficient depending on your specific filesystem.

Where Next?

Popular in Questions Top

bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

Other popular topics Top

aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New

We're in Beta

About us Mission Statement