fireproofsocks

fireproofsocks

S3 operations returning HTTP 505 error

Given a particularly devious S3 object key, e.g.

"users/f8db13a0-e2b8-45cd-a902-5eede074eb46/~*\\|:\"<> +`!@#$%^&()-=_[]{};',.?🧪//fakedata~*\\|:\"<> +`!@#$%^&()-=_[]{};',.?🧪.txt"

I’m running into HTTP 505 errors from the ExAws client, e.g.

    bucket
    |> ExAws.S3.head_object(object_key)
    |> ExAws.request()

I can see the objects there if I run ExAws.S3.list_objects/2 or ExAws.S3.list_objects_v2/2

A python script uploaded the file (probably using official boto), but I’m failing to retrieve it.

Frustratingly, the AWS docs about 5xx errors do not mention 505 errors anywhere. There are few other mentions of this error (e.g. this one).

Because I’m able to interact with other ā€œnormally namedā€ objects in the same bucket, the culprit is likely the weird object key. Has anyone else encountered this? Is this perhaps a bug in ExAws.S3?

Most Liked

evadne

evadne

The proper fix/workaround is to store objects with GUID keys in S3 only (actual file name stored in database) and then use content-disposition to emit correct file names later.

As @LostKobrakai discussed, you can consider sanitising/encoding the characters or changing them to hex representation, however there is no standardised way to pass string literal within JSON & there may be significant work required to handle these files.

I think only place in ex_aws to change would be to do this encoding/decoding transparently (as breaking change) in ExAws.Operation.S3.add_bucket_to_path/2 or maybe ratchet up a step and use XML for outgoing requests (so you get proper string literals that can represent the object names you need), however as it would create such huge support burden for the maintainers + is already discouraged by AWS + would encourage further bad application design, I don’t think it will be considered AT ALL

So my best suggestion is to change design as previously mentioned to obviate the need of this rigmarole. It is also the right thing to do, if you could sanitise the names with some kind of URI encoded string and avoid a whole class of security bugs… such as user putting a file name that effectively results in a path traversal exploit. The bigger the attack surface the more bigger your risk.

Where Next?

Popular in Questions Top

yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
script
If I have a string ā€œ1000 cfu/mlā€ . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ā€˜elixir’ comma...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement