hello, i have the following problem:

I need to make http requests for a given site with thousands of pages. just stop the requests when the response returns {_page,}

I was able to make the code without using concurrency, and Tesla for requests

def fetch(page) do
   {:ok, response} = get(“/api/res”, query: [page: page])
   {page, response.body[“res”]}
end

def fetchAll(curr \\ 1 , list \\ ) do
  result = fetch(curr)
  case result do
   {_page, } →
      list
   {curr, nil} →
      fetchAll(curr,list)
   {_page, res} →
       curr = curr + 1
       fetchAll(curr,list++res)
  end
end

however I wanted to use Task to increase speed, but i don’t know how to do it.

if the number of pages was known, it would be easy to implement it using Task.async_stream

1…numberPages
|> Task.async_stream(fn page ->fetch(page) end, ordered: false,
max_concurrency: System.schedulers_online()*3)
|> Enum.reduce(, fn {:ok, {_page, res}}, j → res++ j end)

but as the number of pages depends on the return of the request, I don’t know how to do this