fireproofsocks

fireproofsocks

S3 operations returning HTTP 505 error

Given a particularly devious S3 object key, e.g.

"users/f8db13a0-e2b8-45cd-a902-5eede074eb46/~*\\|:\"<> +`!@#$%^&()-=_[]{};',.?đź§Ş//fakedata~*\\|:\"<> +`!@#$%^&()-=_[]{};',.?đź§Ş.txt"

I’m running into HTTP 505 errors from the ExAws client, e.g.

    bucket
    |> ExAws.S3.head_object(object_key)
    |> ExAws.request()

I can see the objects there if I run ExAws.S3.list_objects/2 or ExAws.S3.list_objects_v2/2

A python script uploaded the file (probably using official boto), but I’m failing to retrieve it.

Frustratingly, the AWS docs about 5xx errors do not mention 505 errors anywhere. There are few other mentions of this error (e.g. this one).

Because I’m able to interact with other “normally named” objects in the same bucket, the culprit is likely the weird object key. Has anyone else encountered this? Is this perhaps a bug in ExAws.S3?

Most Liked

evadne

evadne

The proper fix/workaround is to store objects with GUID keys in S3 only (actual file name stored in database) and then use content-disposition to emit correct file names later.

As @LostKobrakai discussed, you can consider sanitising/encoding the characters or changing them to hex representation, however there is no standardised way to pass string literal within JSON & there may be significant work required to handle these files.

I think only place in ex_aws to change would be to do this encoding/decoding transparently (as breaking change) in ExAws.Operation.S3.add_bucket_to_path/2 or maybe ratchet up a step and use XML for outgoing requests (so you get proper string literals that can represent the object names you need), however as it would create such huge support burden for the maintainers + is already discouraged by AWS + would encourage further bad application design, I don’t think it will be considered AT ALL

So my best suggestion is to change design as previously mentioned to obviate the need of this rigmarole. It is also the right thing to do, if you could sanitise the names with some kind of URI encoded string and avoid a whole class of security bugs… such as user putting a file name that effectively results in a path traversal exploit. The bigger the attack surface the more bigger your risk.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

Other popular topics Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

We're in Beta

About us Mission Statement