Flyers

Flyers

Zip file name encoding

Hi,

I’m trying to unzip an uploaded file and store the compressed files to my database. I’ve a working process but I’ve just discovered that files with non ascii chars in names are stored weirdly in my postgres database.

How I unzip and save file is :

with {:ok, handle} <- :zip.zip_open(path, [{:cwd, path_name}]),
      {:ok, file_names} = :zip.zip_get(handle) do
  try do
    filter_hidden_files(file_names)
    |> to_plug_upload()
    |> to_multi(user, kind, Multi.new())
    |> Repo.transaction()
  after
    :zip.zip_close(handle)
  end
end

defp filter_hidden_files(_, files \\ [])

defp filter_hidden_files([], files) do
  files
end

defp filter_hidden_files([file | tail], files) do
  filename = Path.basename(file)

  case String.first(filename) === "." do
    true -> filter_hidden_files(tail, files)
    false -> filter_hidden_files(tail, files ++ [file])
  end
end

defp to_plug_upload(_, uploads \\ [])

defp to_plug_upload([], uploads) do
  uploads
end

defp to_plug_upload([file | tail], uploads) do
  upload = %Plug.Upload{
    content_type: MIME.from_path(file),
    filename: Path.basename(file),
    path: to_string(file)
  }

  to_plug_upload(tail, uploads ++ [upload])
end

defp to_multi([], %User{}, _kind, multi) do
  multi
end

defp to_multi(
        [file | tail],
        %User{entity_name: entity_name} = user,
        kind,
        multi
      ) do
  attrs = %{
    brand: entity_name,
    name: file,
    kind: kind,
    file_hash: file_hash(file),
    file_name: file.filename
  }

  m =
    %Media{}
    |> Media.changeset(attrs)

  to_multi(tail, user, kind, multi |> Multi.insert(file.filename, m))
end

When I try to upload and save a file called 2809575_xxléàî.jpg I get the following saved in database 2809575_xxléàî.jpg?63728256310

I think there is an encoding problem when reading files using erlang :zip module.

Thanks.

Most Liked

NobbZ

NobbZ

As far as I know, ZIP does not enforce any encoding of filenames, it is just saved as it comes from the operating system as raw bytes.

So how well non-asciis get carried over depends on where you zip and where you unzip, but as one does not use non-ascii in filenames, this in not a problem at all :wink:

If you know the encoding used by the “zipper”, then you should be able to use iconv like tools to transfer in any encoding you like.

Also remember, that you get double trouble if the encoding in the database differs from the encoding of data you write to it.

Where Next?

Popular in Questions Top

LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

We're in Beta

About us Mission Statement