cipher
Fetch data from multiple pages
how do I make my code get the number of pages available in advance, and then start several tasks to get several pages concurrently?
Example: example.com/api?id=1; example.com/api?id=2; …
and get the response from the server, until the page returns an empty array: “[]”
Most Liked
kokolegorille
There is this old post, I remember I liked it, but maybe now it’s a bit old…
Maybe You can get some inspiration. It uses poolboy and HTTPoison.
Because the problem is not really to be concurrent, but to manage this concurrency.
You don’t want to flood the host with 10’000 concurrent requests.
benwilson512
Hey cipher welcome! Any good programming task starts by breaking down a problem into simpler parts. The first part is: How to fetch a single page? What have you tried so far?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"