tanweerdev

tanweerdev

For no obvious reasons, my request hangs when I try to upload hundreds of images/files to aws

When request hangs control gained back is not printed(when I try to upload files bigger than 20 mb) But I see below error in my console

[warn] ExAws: HTTP ERROR: :closed for URL: "https://logs.us-east-2.amazonaws.com/" ATTEMPT: 1

I use below code to upload files

res =  files_list
    |> Task.async_stream(&upload_file(&1, images_upload), max_concurrency: 4, timeout: 1_500_000)
    |> Stream.run()
IO.inspect "control gained back"
res

and

def upload_file({file_type, local_file_path, file_name}, %ImagesUpload{} = images_upload) do
    file = File.read!(local_file_path)
    destination_path = Path.join([images_upload.upload_id, @file_types[file_type], file_name])

    S3.put_object(component_bucket(), destination_path, file)
    |> ExAws.request!()

    Logger.info("uploaded image #{destination_path}")
    {:ok, destination_path}
end

and in postman I get msg

Could not get response
Error: socket hang up

Most Liked

Eiji

Eiji

@tanweerdev Connection could be closed for various reasons more or less related with client. However if you did not reached any edge-case I think that you hit some timeout. Did you tried changing hackney options? Did you tried retries configuration?

I guess you have just started using this library or some configuration you have is still default (ATTEMPT: 1 so most probably no retries configuration). For bigger files and especially for slow internet connection chance for timeout may be really big, so it’s important that proper download/upload strategy is used regardless of service/software you use as everything have limits.

Also is your internet stable? I don’t think it’s that, but it’s just a good example … Some time ago fiber network I’m using was overloaded (people stay at home - read: “facebook” - because of covid restrictions) and I had many connection problems that time.

Where Next?

Popular in Questions Top

gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement