It uses the default demand dispatcher from GenStage. Can you please post your topology definition? A high number of workers and low max_demand should be enough to make it process one by one. Also, you may be running into the issue that the producer cannot produce fast enough, so there is not enough work for all cores. More information would be appreciated.

Yes, apparently I can’t internet today.

Apologies, I only check the forum a couple times per week. Let’s move this to a Broadway issue and discuss there. But one issue is for sure: if you have multiple producers, the first processor would subscribe to all of them and be the first in the list to receive messages from all of them. The minimum we can do here is to shuffle the list of subscription. I have some ideas to try out, please open up an issue on Broadway and we can continue the convo there!

Here is the broadway setup I tried in the above screenshot. Previously I tried 200 workers with the same effect. I can try 20k but I definitely don’t want to be processing that many files at a time. The files size range unfortunately from a few kb to 200-300 MB so their weights are probably not being respected.

    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producers: [
        default: [
          module:
            {BroadwaySQS.Producer,
             queue_name: "my-pipeline-prototype",
             max_number_of_messages: 10,
             wait_time_seconds: 20,
             visibility_timeout: 300,
             receive_interval: 10},
          stages: 50
        ]
      ],
      processors: [
        # downloaders: [
        #   max_demand: 1,
        #   stages: System.schedulers_online()
        # ],
        default: [
          max_demand: 1,
          # + div(System.schedulers_online(), 2)
          stages: 2000
        ]
      ],
      batchers: [
        default: [
          batch_size: 10,
          batch_timeout: 10_000,
          stages: 50
        ]
      ]
    )

There is more information in my last post:

Yeah, I think the issue is that we can’t get that much data that fast. Try starting two instances of the same broadway (with different names) and see if you can max it out this way.