aura999

aura999

Issue with streaming into a csv

I am trying to stream into a csv file using nimble_csv(1.2.0) but running into issues.

This is the code I am using to create a csv file from the stream.

defmodule Utils
  alias NimbleCSV.RFC4180, as: CSV

  def write_to_file(content_stream) do
    path = Path.join(System.tmp_dir(), "#{UUID.uuid4(:hex)}.csv")

    content_stream
    |> CSV.to_line_stream()
    |> CSV.parse_stream([skip_headers: false])
    |> Stream.into(File.stream!(path, [:write, :utf8]))
    |> Stream.run()
  end
end

This is my test data

contents = [
    "First Name,Last Name,Email\n",
    "David,Byrne,david@test.com"
  ]
  |> Stream.map(&String.trim_leading/1)
   
Utils.write_to_file(contents)

The file is created however the contents in the file are missing the comma separator
and the new line characters

First NameLast NameEmailDavidByrnedavid@test.com

If however I update Utils.write_to_file/1 to by removing parse_stream/2

def write_to_file(content_stream) do
    path = Path.join(System.tmp_dir(), "#{UUID.uuid4(:hex)}.csv")

    content_stream
    |> CSV.to_line_stream()
    |> Stream.into(File.stream!(path, [:write, :utf8]))
    |> Stream.run()
end

I get the csv contents in the correct format.

First Name,Last Name,Email
David,Byrne,david@test.com

I am not sure if I am missing any options that I need to pass to parse_stream/2 that would cause the contents to be malformed.

Any help would be appreciated

Marked As Solved

LostKobrakai

LostKobrakai

Streams are lazy enumerables. Being a stream doesn’t tell you anything about what data the streams deals with. parse_stream maps from a stream of csv formatted binaries to a stream of rows being lists of cell contents. It’s the lazy version to parse_enumerable, which immediately returns [[binary()]].

In your case I’m really confused though because if you already have a csv file and you want to write that csv file, then there’s no need for NimbleCSV in the first place.

Enum.into(content_stream, file_collectable) would be all you need.

What makes streams even more confusing here is that a File.Stream struct implements both Enumerable.t (a - in this case lazy – collection to enumerate) as well as Collectable.t (a collection, which items can be pushed into), which work independently. You’re using the latter functionality here.

Where Next?

Popular in Questions Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

We're in Beta

About us Mission Statement