tanweerdev
For no obvious reasons, my request hangs when I try to upload hundreds of images/files to aws
When request hangs control gained back is not printed(when I try to upload files bigger than 20 mb) But I see below error in my console
[warn] ExAws: HTTP ERROR: :closed for URL: "https://logs.us-east-2.amazonaws.com/" ATTEMPT: 1
I use below code to upload files
res = files_list
|> Task.async_stream(&upload_file(&1, images_upload), max_concurrency: 4, timeout: 1_500_000)
|> Stream.run()
IO.inspect "control gained back"
res
and
def upload_file({file_type, local_file_path, file_name}, %ImagesUpload{} = images_upload) do
file = File.read!(local_file_path)
destination_path = Path.join([images_upload.upload_id, @file_types[file_type], file_name])
S3.put_object(component_bucket(), destination_path, file)
|> ExAws.request!()
Logger.info("uploaded image #{destination_path}")
{:ok, destination_path}
end
and in postman I get msg
Could not get response
Error: socket hang up
Most Liked
Eiji
@tanweerdev Connection could be closed for various reasons more or less related with client. However if you did not reached any edge-case I think that you hit some timeout. Did you tried changing hackney options? Did you tried retries configuration?
I guess you have just started using this library or some configuration you have is still default (ATTEMPT: 1 so most probably no retries configuration). For bigger files and especially for slow internet connection chance for timeout may be really big, so it’s important that proper download/upload strategy is used regardless of service/software you use as everything have limits.
Also is your internet stable? I don’t think it’s that, but it’s just a good example … Some time ago fiber network I’m using was overloaded (people stay at home - read: “facebook” - because of covid restrictions) and I had many connection problems that time.








