alphydan
Error when importing CSV with Explorer.DataFrame.from_csv! possibly related to dtype
Hello,
I’m an elixir beginner, exploring livebook. I’m running a local notebook on a linux machine. Initially I tried to import a CSV file with Explorer like so:
alias Explorer.{DataFrame, Series}
path_to_csv = "./some_file.csv"
DataFrame.from_csv!(path_to_csv)
However, I am seeing the following error:
** (RuntimeError) from_csv failed: {:polars, "Could not parse `Fee Amount`
as dtype Int64 at column 16.\nThe current offset in the file is 249 bytes.\n\n
Consider specifying the correct dtype, increasing\n
the number of records used to infer the schema,\n
enabling the `ignore_errors` flag, or adding\n`Fee Amount` to the `null_values` list."}
(explorer 0.5.5) lib/explorer/data_frame.ex:438: Explorer.DataFrame.from_csv!/2
To check if the problem was indeed with the data in the column Fee Amount, I filled the column with integers. But I get the same error. What can I try to better diagnose this problem?
Most Liked
BradS2S
Oh yes the error message could be more helpful but it is a parsing error. It sees the newline character as a new row. It looks forward and sees mostly numbers in the row so it decides to infer int64 which doesn’t work for the second half of the string.
LostKobrakai
I’m totally fine with headers be required to be single line. I’m just argueing that this should not be a CSV parsing error, but a proper “hey we need you to have single line headers” error.
benwilson512
al2o3cr
One small thing that jumps out:
Could not parse
Fee Amountas dtype Int64 at column 16.
The current offset in the file is 249 bytes.
This sounds like it’s on at least the second line of the file, and so should be data
That’s a reference to the with_null_values setting in polars_io::csv::CsvReader, which is intended to parse values like "n/a" to null.
The message suggests that the reader found the literal string “Fee Amount” where it expected data.
what was a header-shaped value like “Fee Amount” doing on the second line of a CSV?
alphydan
And yet, pandas can process \n in the header or in the fields if they are quoted.
The csv:
"Timestamp (ISO)",Animal Type,Animal Color,"Friends $
of Given Animal","Another Timestamp$
(ISO)",Feedstock,"Favorite food for Animal",Feed Amount,Feed Frequency^M$
2023,"Tiger and $
Beasts",Yellow,Lion,2023-03-14,3,meat,32.2,weekly^M$
is imported by pandas as:
Timestamp (ISO) Animal Type Animal Color Friends \n of Given Animal Another Timestamp\n (ISO) Feedstock Favorite food for Animal Feed Amount Feed Frequency
0 2023 Tiger and \nBeasts Yellow Lion 2023-03-14 3 meat 32.2 weekly
1 2023 Tiger Yellow Lion 2023-03-14 3 meat 32.2 weekly
Should I give feedback to the Explorer library?
Popular in Questions
Other popular topics
Latest Livebook Threads
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers. By developers, for developers.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"