docjazither

docjazither

No function clause matching in CSV.Decoding.Preprocessing.Lines.starts_sequence?/5

Hey OGs,
I have a function that loads csv files into my app. When my user used special characters, it failed
they used name like: Zoë
Here is my function:

defp read(csv) do
    csv.path
    |> Path.expand()
    |> File.stream!()
    |> CSV.decode(headers: true)
    |> Stream.map(fn {:ok, data} -> data end)
    |> Stream.reject(&is_nil/1)
    |> break_point()
    |> Enum.to_list() # It fails here
    |> Enum.uniq()
  end

Error:

The following arguments were given to CSV.Decoding.Preprocessing.Lines.starts_sequence?/5:
    
        # 1
        <<145, 32, 72, 111, 97, 100, 44, 105, 110, 115, 116, 97, 103, 114, 97, 109>>
    
        # 2
        "o"
    
        # 3
        false
    
        # 4
        44
    
        # 5
        ""
Attempted function clauses (showing 5 out of 5):
    
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, last_token, false, separator, _) when last_token == <<separator::utf8()>>
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, "", false, separator, _)
        defp starts_sequence?(<<34::utf8(), tail::binary()>>, _, quoted, separator, sequence_start)
        defp starts_sequence?(<<head::utf8(), tail::binary()>>, _, quoted, separator, sequence_start)
        defp starts_sequence?("", _, quoted, _, sequence_start)

Apparently, it couldn’t recognise this character ë
I wished I could like make a callback for starts_sequence? to return a meaningful error or something like that…
How can I prevent this from happening ?

Marked As Solved

al2o3cr

al2o3cr

Make sure you have the latest version of csv - version 2.5.0 contains a fix for this error:

Also Liked

al2o3cr

al2o3cr

The offending byte is the first one of the binary in the first argument of that error - 145. The preceding character was apparently o (the second argument to starts_sequence?) so a byte with value 145 is not valid UTF8.

145 happens to be a left single-quote () in Windows-1252, which is probably what the creator of the CSV intended.

There’s no good way to handle this error apart from upgrading, as the code that was added back in 2.5.0 is the error handling code!

al2o3cr

al2o3cr

AFAIK the only currently available way to deal with malformed UTF8 in CSV.decode is to pass the undocumented replacement option - it’s passed down unmodified to CSV.Decoding.Decoder where it is documented.

Prior to 2.5.0, this wasn’t an option because replacement happens in the lexer way after the starts_sequence? code in the preprocessor, so bad characters never made it to where they could be replaced.

A possible extension to CSV would be to also accept a 1-argument function as replacement, and provide it with the bad character.

Where Next?

Popular in Questions Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
beno
I will often find my self writing things similar to: case some_value do nil -&gt; something() "" -&gt; something() _ -&gt; someth...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

We're in Beta

About us Mission Statement