FWIW - I recently changed the library over to match the community guidelines for Broadway by renaming to off_broadway_redis.

Note the namespace changes. Outside of that, its a drop-in replacement for the OG version.

Just to add a little more detail on the guts of this - for those that don’t want to dig through the README.

• My implementation follows the ‘Reliable Queue’ pattern outlined in the Redis documentation (https://redis.io/commands/rpoplpush#pattern-reliable-queue). This means it relies on an RPOPLPUSH command to atomically move the item from the source list to a target ‘working’ or ‘processing’ list before returning said item to the producer. So be sure you push items to the left side of the Redis list you are working from (using LPUSH) - this assumes you doing FIFO work.
• I handle the acknowledgement aspect of Broadway by batching Redis LREM commands to the working list.
• Both Redis commands (RPOPLPUSH and LREM) are pipelined with sensible default batch sizes and limits.

Biggest caveat now is that you must supply the supervised Redis connection atom to the producer when starting. Maybe not a big deal, but I didn’t want to manage that in the initial release and keep it as simple as possible.

I’m no longer crawling/scraping the web, but I do use the Broadway library for enterprise data integrations now. Although it is not all that different - except customers pay to access an endpoint they provide.