fchabouis
IO.binstream is not unicode safe. What does it mean?
Hello,
I have read the IO module documentation and I must say I am confused about it and all the types related to strings.
In particular, I read for the binstream functio doc that is is “Unicode unsafe”. And that “Finally, do not use this function on IO devices in Unicode mode as it will return the wrong result.”
Now if I try:
{:ok, pid} = "àéb€%$\nùç*µ" |> StringIO.open()
> {:ok, #PID<0.114.0>}
pid |> IO.binstream(:line) |> Enum.to_list()
> ["àéb€%$\n", "ùç*µ"]
It seems to work. Is that just lucky ?
If I create to IO device with StringIO.open(), I understand it is a unicode IO device. Am I supposed to stream it with IO.stream instead of IO.binstream? Does it make a difference?
Thanks!
PS: I read for binstream that “The device is iterated by the given number of bytes or line by line if :line is given. This reads from the IO device as a raw binary.” What’s a raw binary ? Binary data with non-unicode encoding for example? But how can binstream detect new lines if the encoding is unknown?
Most Liked
josevalim
\n might work by accident because in almost all encodings it is the 10 byte. The issue with binstream will manifest in cases where the device supports multiple encodings, such as files. Try this:
- Write Unicode to a file
- Open the file with :utf8 flag
- Read the file with binread
In a nutshell, if a file was opened without the utf8 flag, use binread, otherwise read.
PS: yes, a raw binary is meant to be a string that is not in utf8 encoding. Improvements to the docs are welcome!
fchabouis
Thanks for the reply José.
I did the test you suggested, and using binstream on a file opened with :utf8 effectively gives wrong results.
From what I understand, the doc in binstream implicitely refers to the doc of File.open. But in my case, the input did not come from a File but from a string. Thus I was reading the documentation of StringIO.open, which also has an encoding option.
When the IO device comes from StringIO.open, no matter what encoding (:unicode or :latin1) is chosen, or wich function is used to stream (IO.binstream or IO.stream), everything seems to work fine.
Thanks for your help!
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"