pelopo

pelopo

Read compressed files as string and not bitstring

Hi, I’m reading a file compressed originally with LZMA and ext .txt.xz. I can read it with stream ok, but the output is a bitstring and not a string. How can I convert it to string. The search didn’t give me any meaningful solutions. Thanks

My func

defmodule FileCounter2 do
  def words_per_line!(path) do
    File.stream!(path, [:read, :compressed], :line)
    |> Stream.map(&String.split/1)
    |> Enum.each(&IO.inspect/1)
  end
end
file = "0001008629-99-000012.txt.xz"
FileCounter2.words_per_line!(file)

My output

[
  <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0,
    116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>,
  <<136, 164, 132, 123, 157, 76, 98, 215, 159, 145, 220, 87, 168, 219, 206, 16,
    211, 225, 94, 191, 230, 193, 54, 16, 26, 112, 245, 120, 145, 191, 25, 244>>,
  <<76, 153, 192, 109, 179, 201, 164, 2, 175, 5, 63, 240, 147, 190, 60, 26, 97,
    253, 81, 38, 245, 131, 209, 93, 119, 109, 57>>
]
[
  <<96, 188, 128, 77, 17, 236, 2, 66, 162, 253, 103, 160, 100, 1, 108, 37, 216,
    25, 220, 55, 21, 177, 158, 216, 91, 37, 123, 195, 164, 89, 75, 186, 133, 97,
    66>>,
  <<229, 135,.......]

Most Liked

kip

kip

ex_cldr Core Team

Strings in Elixir are, by definition, UTF-8 encoded.

When you inspect a binary (a binary is a bitstring that is divisible by 8 bits - therefore a String is also a binary encoded as UTF-8), Elixir will output it as a string if it is valid UTF-8. It will output it as a binary if it is not.

Therefore basically your data is not valid UTF-8 and is therefore not a String.

You can chunk the binary by the valid String parts as follows:

iex> binary = <<253, 55, 122, 88, 90, 0, 0, 4, 230, 214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47, 229, 163, 224, 46, 175, 14, 134, 93, 0, 22, 233>>
iex> String.chunk(binary, :valid)
[
  <<253>>,
  <<55, 122, 88, 90, 0, 0, 4>>,
  <<230>>,
  <<214, 180, 70, 2, 0, 33, 1, 22, 0, 0, 0, 116, 47>>,
  <<229, 163, 224>>,
  ".",
  <<175>>,
  <<14>>,
  <<134>>,
  <<93, 0, 22>>,
  <<233>>
]

That suggests your data is encoded in some very different way to UTF-8.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I think the important thing here is that there’s no singular way to compress a file, rather there are dozens and dozens of different compression algorithms out there. You need to use a program or library that implements the specific decompression algorithm for the file that you have. There are a couple that have a built in implementation inside erlang (and therefore Elixir) zlib — erts v15.0.1. I don’t think however that xz compression is the same as gzip.

Where Next?

Popular in Questions Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
beno
I will often find my self writing things similar to: case some_value do nil -&gt; something() "" -&gt; something() _ -&gt; someth...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
New
beno
I will often find my self writing things similar to: case some_value do nil -&gt; something() "" -&gt; something() _ -&gt; someth...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

We're in Beta

About us Mission Statement