preciz

preciz

How to properly convert this ?Octal? encoded UTF16 BE string to UTF8

This is the string:

"\\376\\377\\0001\\0009\\000 \\0001\\0002\\000 \\0001\\0009\\000 \\000P\\000r\\000o\\000g\\000r\\000a\\000m\\000m\\000f\\000o\\003\\b\\000r\\000d\\000e\\000r\\000u\\000n\\000g\\000 \\0002\\0000\\0002\\0000\\000 \\000-\\000 \\000A\\000n\\000l\\000a\\000g\\000e\\000 \\0002\\000 \\000-\\000 \\000F\\000o\\003\\b\\000r\\000d\\000e\\000r\\000e\\000r\\000g\\000e\\000b\\000n\\000i\\000s\\000s\\000e\\000.\\000d\\000o\\000c\\000x"

That should be converted to: "19 12 19 Programmförderung 2020 - Anlage 2 - Förderergebnisse.docx"

It’s from a PDFs /Info dictionary.

The "\\376\\377" is a BOM (byte order mark) meaning it’s an UTF16 big endian.

Firefox can read it properly and I assume this is the source code they use:

I did try to replicate this in Elixir but when it gets to the "ö" it breaks into non relevant characters.

What would be the correct way to convert the string?

Marked As Solved

malaire

malaire

That shown code isn’t enough to decode the string as it doesn’t handle decoding \\003\\b to UTF16 codepoint 0x0308.

This could be fixed e.g. by first converting \\b to \\010, but full solution would need a list of all possible escape codes used in the encoding.

Also Liked

malaire

malaire

It looks like UTF16 BE bytes in printable ASCII range are represented as ASCII characters (e.g. P instead of \\120), and at least some UTF16 BE bytes in ASCII control character range use the usual escape codes (e.g. \\b meaning ASCII backspace).

So encoding might also be using some other escapes like \\n for \\012 (line feed) or \\r for \\015 (carriage return).

g-andrade

g-andrade

This seems to do it in Erlang:

convert_text_with_bom_to_utf8() ->
    Data = <<"\376\377\0001\0009\000 \0001\0002\000 \0001\0009\000 "
             "\000P\000r\000o\000g\000r\000a\000m\000m\000f\000o\003"
             "\b\000r\000d\000e\000r\000u\000n\000g\000 \0002\0000"
             "\0002\0000\000 \000-\000 \000A\000n\000l\000a\000g"
             "\000e\000 \0002\000 \000-\000 \000F\000o\003\b\000r\000d\000e\000r"
             "\000e\000r\000g\000e\000b\000n\000i\000s\000s\000e"
             "\000.\000d\000o\000c\000x"
           >>,

    {Encoding, BomLength} = unicode:bom_to_encoding(Data), % determine Data encoding from BOM
    <<_:BomLength/bytes, Text/bytes>> = Data, % thrown BOM away
    <<_/bytes>> = unicode:characters_to_binary(Text, Encoding, utf8). % convert to UTF-8

You can interact with it here.

Unfortunately I wasn’t able to have declare an equivalent literal value in Elixir; but, as long as your string is in the expected format, the same pattern should work.

If, however, the double-backslashes represent actual backslashes in the original string and are not the result of printing / inspecting it, then they’ll need to be unescaped first.

g-andrade

g-andrade

Now, as far as unescaping the string - if needed - this seems to do the trick:

  def unescape(data) do                                                                                                                                                                                                                        
    charlist = String.to_charlist(data)                                                                                                                                                                                                        
    erlang_literal = '"#{charlist}"'                                                                                                                                                                                                           
    {:ok, [{:string, _, unescaped_charlist}], _} = :erl_scan.string(erlang_literal)                                                                                                                                                            
    List.to_string(unescaped_charlist)                                                                                                                                                                                                         
  end  

What it does is transform the string into a charlist containing an Erlang “string expression”, which it proceeds to parse - and so I wouldn’t trust it with untrusted input without further investigation.

hauleth

hauleth

You should check :unicode module for that. It has all functions you will need.

Where Next?

Popular in Questions Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

We're in Beta

About us Mission Statement