Questions & Help>Questions

HTTPotion/:ibrowse slows down under load

#performance#httpotion
adobroskok

adobroskok

HTTPotion/:ibrowse slows down under load

In my project I spawn separate process for each download with HTTPotion, which works perfect on small download numbers (<5), but if I try to push higher load (~100 files) performance degrades substantially - from 20s per file to 900s per file and I start to get :timeout errors. I tried to tune performance via :ibrowse options, but didn’t get much improvement.
I have put together a project that demonstrates the issue: https://github.com/andre-dubber/download_issue
Is there a way keep high throughput without distributing load over multiple Erlang nodes?

P.S. I did create a separate branch in my project where I don’t use spawn as recommended by HTTPotion author, but behaviour is still same.

Any suggestions?

2 comments
#httpotion
1 741 2

First Post!

dimitarvp

dimitarvp

I know this is probably not what you are looking for but, have you tried using :httpc or HTTPoison (which uses :hackney)?

Alternatively, better use Tesla and switch underlying HTTP client for easier testing and evaluation.

Post #2

Where Next?

View thread on forum (has 2 responses!)
performance
httpotion
Home Questions & Help>Questions
#performance#httpotion
1 742 2
Last post

Popular in Questions Top

senggen
Questions & Help>Questions

How to fix *Bad argument in call to erlang:'++'(<<"xxx/crash.log">>, ".3") in lager_rotator_default:rotate_logfile/2 line 84*
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
#production#error#log
2 46314 2
New
bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 22889 6
New
logicmason
Questions & Help>Questions

What's the best way to serve restricted ports (e.g. 80, 443) with Phoenix?
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
/phoenix#deployment#linux
32 16824 13
New
nsuchy
Questions & Help>Questions

IEX in Windows Powershell?
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
#iex#microsoft-windows#windows#powershell
15 29309 4
New
vac
Questions & Help>Questions

Formatting a string to PEM with headers and line breaks
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
2 14555 7
New
minhajuddin
Questions & Help>Questions

Using List.first instead of Enum.at(0)
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
#code-style
76 33273 14
New
alice
Questions & Help>Questions

What are the main benefits of Elixir compared to Clojure?
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
#clojure
223 32979 47
New
shahryarjb
Questions & Help>Questions

How to convert map to string (separated with ,)
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
#maps#strings
15 15491 2
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
sabri
Questions & Help>Questions

Ecto pool_size?
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
#ecto
30 14050 5
New

Other popular topics Top

Qqwy
Blogs & Podcasts>Blog Posts

Elixir Blog Posts
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
hexdocs.pm
#blog-posts#wiki#stickies#official-blog-posts-thread
3268 119930 1237
New
senggen
Questions & Help>Questions

How to fix *Bad argument in call to erlang:'++'(<<"xxx/crash.log">>, ".3") in lager_rotator_default:rotate_logfile/2 line 84*
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
#production#error#log
2 46314 2
New
pmjoe
Questions & Help>Questions

What do you think of Gleam compared to Elixir?
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
#programminguages#gleam
24 16555 10
New
grych
Your Libraries & Projects>Libraries

Drab: remote controlled frontend framework for Phoenix
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
/phoenix#web-frameworks#frontend#javascript#drab
639 49522 488
New
KronicDeth
Libraries & Projects>Libraries

IntelliJ Elixir - Elixir plugin for JetBrain's IntelliJ Platform
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
#intellij
289 35421 110
New
SoCreat
Questions & Help>Questions

What's the best ide/editor for elixir in 2021?
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
#ide
68 37349 27
New
vertexbuffer
Questions & Help>Questions

Deleting item from a list
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
7 24097 4
New
danschultzer
Libraries & Projects>Libraries

Pow: Robust, modular, extendable user authentication and management system
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
/phoenix#authentication#library#authorization#user_system#pow
548 27727 240
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
siddhant3030
Questions & Help>Questions

How can I write a raw sql query?
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
/phoenix#ecto
13 19061 20
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

WyewWorks

Enabling companies to succeed by building software people love.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement