I just had this issue and solved it by specifying the encoding to File.stream!:

    file_path
    |> File.stream!([{:encoding, :latin1}])
    |> CSV.decode(headers: true)
    # ....

I didn’t initially know how the file was encoded; I just used trial-and-error with the various supported encodings listed here.

Thank you for checking that. I tried copying the á and pasting it into a file. This worked great! Unfortunately, I think the issue is that my source file is incorrectly encoded – or I just don’t know how it’s encoded or how to tell Elixir how to read it.

In the “bad” file, it is encoded as a single byte 0xE1 This file has á\n

$ od -x BAD.CSV
0000000 0ae1
0000002

When I copied the character from my browser into a text file, the character is 2 bytes, 0xC3A1 and a \n (0x0a)

od -x good.csv 
0000000 a1c3 000a
0000003

This page has better information on encodings for this character. It seems that good.csv is actually UTF-8 encoded, while bad.csv is something else.

Any thoughts on how to read this mangled encoding?
Either that or I need to find out at which point in the pipeline this file is getting munged.

Wow, I know having clean data is good, but this really drives it home.

I ended up performing this, which feels really hackish

for <<a <- File.read!("BADFILE.CSV") >>, into: <<>>, do: <<a::utf8>>

especially since this returns the whole file as a string so I had to use String.split/2 in order to get it into a format that the CSV library could use.

Looking at this, there’s probably some way to use pattern matching to get the comprehension to output a line at a time, but I’m too tired to work on it right now. It would also be nice if I could use a Stream for the comprehension.

Maybe you already tried this, but can you use File.stream!/3?

Unfortunately, there’s no straightforward way to detect the character encoding of a file. This project called UniversalDetector does some heuristics and statistical analysis to guess the character encoding.

It may be that your file just has junk characters. You could run it through some regex to strip any non-printable characters.

No, File.Stream!/3 won’t work – according to José Valim’s answer here.

I have proposed a “stream for” and “parallel for” for future Elixir versions, however it is pending some other improvements to the language.

I am excited to see those things added, though!

As far as detecting, I already know that these files have poorly encoded characters, they are valid, just not UTF-8. I’m currently mulling over pre-processing the files using tr or something to convert all the odd characters to proper UTF-8 encodings.