Rakakku

Rakakku

Library for exporting CSV

I need to read data from a database, process and transform it and send to a user in CSV.

I’ve found 2 libraries for far: csv and nimble_csv. Both look decent.

Among those 2 or among others, what would you recommend?

Or maybe I don’t need one at all?

Most Liked

michalmuskala

michalmuskala

In my experience nimble_csv should be significantly faster.

lkuty

lkuty

I confirm this from a bad behaviour I experienced with csv. The time taken to dump a CSV file (600 rows, 574 columns) went from ±210s with csv to ±430ms with nimble_csv. It looks like it could be the fault of protocols not being consolidated or the fact that we use the default implementation for Any instead of specialized ones for the built-in types (Integer, Float, …) which are not provided by csv (except for strings). See closed issues: CSV.encode seems extremely slow and Running in production/release slow. I don’t know what is the problem and I do not have the time for the moment to explore the matter.

EDIT
I found the time to make some tests :slight_smile: The protocols are consolidated. The problem comes from the default protocol implementation for Any which uses the implementation for BitString which does extra work for strings and that work is not necessary for simpler types. I have provided implementation for Integer, Float, Atom (for true and false), DateTime and it was quite fast. Going from ±210s to 430ms too :+1:.

defimpl CSV.Encode, for: Integer do
  def encode(data, _env \\ []) do
    data |> Integer.to_string
  end
end
defimpl CSV.Encode, for: Float do
  def encode(data, _env \\ []) do
    data |> Float.to_string
  end
end
defimpl CSV.Encode, for: Atom do
  def encode(data, env \\ [])
  def encode(true, _env), do: "1"
  def encode(false, _env), do: "0"
  def encode(data, _env) do
    data |> Atom.to_string
  end
end
defimpl CSV.Encode, for: DateTime do
  def encode(data, _env \\ []) do
    data |> DateTime.to_string
  end
end

NOTE
I should have mentionned that the times written in the answer take into account the production of the data using :ets.select (and its continuation technique) and the export to CSV.

lkuty

lkuty

Since issue 10 Provide default implementation of CSV.Encode where appropriate was closed by mentioning issue 9 CSV.encode seems extremely slow, I added a comment to that issue. I guess the owner has deliberately left the protocol implementation open like that to force us to implement what we want. IMHO it is counter-intuitive for someone who starts using the library for the first time.

Where Next?

Popular in Questions Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

We're in Beta

About us Mission Statement