Rakakku

Rakakku

Library for exporting CSV

I need to read data from a database, process and transform it and send to a user in CSV.

I’ve found 2 libraries for far: csv and nimble_csv. Both look decent.

Among those 2 or among others, what would you recommend?

Or maybe I don’t need one at all?

Most Liked

michalmuskala

michalmuskala

In my experience nimble_csv should be significantly faster.

lkuty

lkuty

I confirm this from a bad behaviour I experienced with csv. The time taken to dump a CSV file (600 rows, 574 columns) went from ±210s with csv to ±430ms with nimble_csv. It looks like it could be the fault of protocols not being consolidated or the fact that we use the default implementation for Any instead of specialized ones for the built-in types (Integer, Float, …) which are not provided by csv (except for strings). See closed issues: CSV.encode seems extremely slow and Running in production/release slow. I don’t know what is the problem and I do not have the time for the moment to explore the matter.

EDIT
I found the time to make some tests :slight_smile: The protocols are consolidated. The problem comes from the default protocol implementation for Any which uses the implementation for BitString which does extra work for strings and that work is not necessary for simpler types. I have provided implementation for Integer, Float, Atom (for true and false), DateTime and it was quite fast. Going from ±210s to 430ms too :+1:.

defimpl CSV.Encode, for: Integer do
  def encode(data, _env \\ []) do
    data |> Integer.to_string
  end
end
defimpl CSV.Encode, for: Float do
  def encode(data, _env \\ []) do
    data |> Float.to_string
  end
end
defimpl CSV.Encode, for: Atom do
  def encode(data, env \\ [])
  def encode(true, _env), do: "1"
  def encode(false, _env), do: "0"
  def encode(data, _env) do
    data |> Atom.to_string
  end
end
defimpl CSV.Encode, for: DateTime do
  def encode(data, _env \\ []) do
    data |> DateTime.to_string
  end
end

NOTE
I should have mentionned that the times written in the answer take into account the production of the data using :ets.select (and its continuation technique) and the export to CSV.

lkuty

lkuty

Since issue 10 Provide default implementation of CSV.Encode where appropriate was closed by mentioning issue 9 CSV.encode seems extremely slow, I added a comment to that issue. I guess the owner has deliberately left the protocol implementation open like that to force us to implement what we want. IMHO it is counter-intuitive for someone who starts using the library for the first time.

Where Next?

Popular in Questions Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New

We're in Beta

About us Mission Statement