Questions & Help>Questions

File encoding detection library

#csv
patryk.it

patryk.it

File encoding detection library

Hi guys!

My question is related to the file encoding topic. I’m working on a feature that allows users to upload a CSV file that might be in different charset. It’s not only about UTF-8 and I need to detect given encoding and then parse data.

In Golang, Python, Java, C++, Objective-C we have UniversalDetector/chardet libraries. Do we have something similar to the Elixir/Erlang usage? I’m stuck with this.

How are you working with the different charsets/encodings in Elixir applications? I know - that’s not the main problem to which was designed this language, but I won’t use erlports or AWS Lambda only for detecting charset. :frowning:

12 comments
#csv
7 2856 12

Most Liked

NobbZ

NobbZ

There is no way to get what you want.

Ask your clients to upload using a specified charset only.

If you see the single byte 0xC4, that could be a latin-1 encoded Ä, though in ISO-8859-5 (kyrillic) it would be as valid as with latin-1 though encoding a different character, the Ф.

Therefore it is impossible to detect the encoding without knowing the content in advance.

What you describe from the other languages, is usually a very dumb heuristic.

  1. Check if the input is valid UTF-8/16/32 by corresponding validators, perhaps even helped by a BOM
  2. If not, fall back to a single byte encoding derived from the hosts local settings
4
Post #5
LostKobrakai

LostKobrakai

nimble_csv supports converting data to utf-8 before parsing / converting from utf-8 before dumping. By default it even ships NimbleCSV.Spreadsheet, which uses utf-16 le to work with excel csv’s. But you’ll need to setup a module per encoding.

1
Post #2
evadne

evadne

The libMagic route is definitely worth considering. I have a library for that. Low profile, but working.

1
Post #10

Where Next?

View thread on forum (has 12 responses!)
csv
Home Questions & Help>Questions
#csv
7 2857 12
Last post

Popular in Questions Top

vonH
Questions & Help>Questions

Why would I choose Elixir as a general purpose programming language?
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
#functional-programming#use-cases
65 34758 13
New
bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 22889 6
New
JorisKok
Questions & Help>Questions

How is it possible to get 2 million websocket connections when you have 65536 available ports?
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
/phoenix
20 17308 4
New
itssasanka
Questions & Help>Questions

Why use `utc_datetime` over `naive_datetime` for Ecto?
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
#ecto#timestamps#datetime#timezones
115 14814 19
New
aalberti333
Questions & Help>Questions

Enum.map over list of key/value pairs with a map as the value
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
#enummap
7 19677 6
New
myronmarston
Questions & Help>Questions

Typespecs - best way to spec keyword lists?
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
#dialyzer
52 14258 3
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
#ecto
22 17588 9
New
baxterw3b
Questions & Help>Questions

Anonymous functions with multiple body
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
19 21396 4
New
belgoros
Questions & Help>Questions

Regex question for hyphen match
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
#learning-elixir
9 12939 10
New
jc00ke
Questions & Help>Questions

Typespec for map w/both required and optional keys
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
#typespecs
27 13396 13
New

Other popular topics Top

Brian
Questions & Help>Questions

Import a module from a file into IEX
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
#iex
35 32139 16
New
shahryarjb
Questions & Help>Questions

How to convert Date to DateTime which Ecto needs
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
#ecto#datetime
15 14989 5
New
sorentwo
News>Announcing

Oban - Reliable and Observable Job Processing
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
/oban#ecto#postgres#job-processing
977 41022 311
New
srinivasu
Questions & Help>Questions

Best Practises for Error handling elixir?
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
16 16061 9
New
aalberti333
Questions & Help>Questions

Enum.map over list of key/value pairs with a map as the value
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
#enummap
7 19677 6
New
KronicDeth
Libraries & Projects>Libraries

IntelliJ Elixir - Elixir plugin for JetBrain's IntelliJ Platform
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
#intellij
289 35421 110
New
chensan
Questions & Help>Questions

(Postgrex.Error) ERROR 42804 (datatype_mismatch): column "" cannot be cast automatically to type integer
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
#ecto
29 13294 4
New
qwerescape
Questions & Help>Questions

How to get the call stack/stack trace at any point in code?
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
53 15772 4
New
fayddelight
Questions & Help>Questions

Installing elixir via asdf shows zsh: command not found: iex
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
#erlang#asdf
44 16425 17
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement