pelopo
Read compressed files as string and not bitstring
Hi, I’m reading a file compressed originally with LZMA and ext .txt.xz. I can read it with stream ok, but the output is a bitstring and not a string. How can I convert it to string. The search didn’t give me any meaningful solutions. Thanks
My func
defmodule FileCounter2 do
def words_per_line!(path) do
File.stream!(path, [:read, :compressed], :line)
|> Stream.map(&String.split/1)
|> Enum.each(&IO.inspect/1)
end
end
file = "0001008629-99-000012.txt.xz"
FileCounter2.words_per_line!(file)
My output
[
<<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0,
116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>,
<<136, 164, 132, 123, 157, 76, 98, 215, 159, 145, 220, 87, 168, 219, 206, 16,
211, 225, 94, 191, 230, 193, 54, 16, 26, 112, 245, 120, 145, 191, 25, 244>>,
<<76, 153, 192, 109, 179, 201, 164, 2, 175, 5, 63, 240, 147, 190, 60, 26, 97,
253, 81, 38, 245, 131, 209, 93, 119, 109, 57>>
]
[
<<96, 188, 128, 77, 17, 236, 2, 66, 162, 253, 103, 160, 100, 1, 108, 37, 216,
25, 220, 55, 21, 177, 158, 216, 91, 37, 123, 195, 164, 89, 75, 186, 133, 97,
66>>,
<<229, 135,.......]
Most Liked
kip
Strings in Elixir are, by definition, UTF-8 encoded.
When you inspect a binary (a binary is a bitstring that is divisible by 8 bits - therefore a String is also a binary encoded as UTF-8), Elixir will output it as a string if it is valid UTF-8. It will output it as a binary if it is not.
Therefore basically your data is not valid UTF-8 and is therefore not a String.
You can chunk the binary by the valid String parts as follows:
iex> binary = <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>
iex> String.chunk(binary, :valid)
[
<<253>>,
<<55, 122, 88, 90, 0, 0, 4>>,
<<230>>,
<<214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47>>,
<<229, 163, 224>>,
".",
<<175>>,
<<14>>,
<<134>>,
<<93, 0, 22>>,
<<233>>
]
That suggests your data is encoded in some very different way to UTF-8.
benwilson512
I think the important thing here is that there’s no singular way to compress a file, rather there are dozens and dozens of different compression algorithms out there. You need to use a program or library that implements the specific decompression algorithm for the file that you have. There are a couple that have a built in implementation inside erlang (and therefore Elixir) zlib — erts v15.0.1. I don’t think however that xz compression is the same as gzip.







