Questions & Help>Questions

Simple crawler using GenStage (questions about the structure)

#worker-pools#genstage#how-to-question
ravernkoh

ravernkoh

Simple crawler using GenStage (questions about the structure)

Hi! I intend to build a simple web crawler in Elixir, and since a web crawler is really just a pipeline of url-queue -> fetcher -> parser (with some filters and rate limiters in between) I have decided to use GenStage for this. I have already built the url queue. Now, I’m thinking of how to build the fetcher.

Obviously, the fetcher will need to perform HTTP requests in parallel, and since there is a limit to how many I should do at a time (bandwidth right…? correct me if I’m wrong about this), I should use some sort of pooling system. I have looked at ConsumerSupervisor in the GenStage docs, but it is only for the last part, the consumer portion.

How should I go about implementing this, where the fetcher stage is a producer_consumer (So i cant use the ConsumerSupervisor)?

2 comments
#worker-pools#genstage#how-to-question
4 929 2

Most Liked

Qqwy

Qqwy

TypeCheck Core Team

If your crawler wants to grow to become more sophisticated, I’d suggest looking into Event Sourcing: By persisting the results of each stage independently, you could at some point decide that the parser can be smarter (such as extract more info) without having to re-crawl all previously-crawled resources.

As for implementing the fetcher stage itself, I think Crawlie’s UrlManager indeed gives some good suggestions :slight_smile: . Great tip, @mischov!

2
Post #3
mischov

mischov

For some ideas you might check out Crawlie, which I believe uses GenStage.

1
Post #2

Where Next?

View thread on forum (has 2 responses!)
worker-pools
genstage
how-to-question
Home Questions & Help>Questions
#worker-pools#genstage#how-to-question
4 930 2
Last post

Popular in Questions Top

bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 22889 6
New
quazar
Questions & Help>Questions

How to encode ecto schema with Jason with all fields
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
/phoenix#json#jason
17 13698 9
New
SoCreat
Questions & Help>Questions

What's the best ide/editor for elixir in 2021?
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
#ide
68 37349 27
New
dokuzbir
Questions & Help>Questions

How to convert list to string without losing brackets
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
26 14426 10
New
vac
Questions & Help>Questions

Formatting a string to PEM with headers and line breaks
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
2 14555 7
New
freewebwithme
Questions & Help>Questions

Failed to run 'elixir' command error in vs code
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
#vscode#elixir-ls
48 15871 39
New
mcarvalho
Questions & Help>Questions

System.get_env vs. Application.get_env
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
#environment
13 17021 6
New
gonzofish
Questions & Help>Questions

Ecto: how to join three tables?
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
/phoenix#ecto
21 13889 7
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
beno
Questions & Help>Questions

Pattern match with multiple match options
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
#pattern-matching#how-to-question
29 15490 6
New

Other popular topics Top

senggen
Questions & Help>Questions

How to fix *Bad argument in call to erlang:'++'(<<"xxx/crash.log">>, ".3") in lager_rotator_default:rotate_logfile/2 line 84*
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
#production#error#log
2 46314 2
New
sergio
Questions & Help>Questions

Updating a field using Ecto one-liner?
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
15 14579 3
New
gshaw
Questions & Help>Questions

Idiomatic guard clause for checking not nil
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
#guard-clauses
13 43178 3
New
KronicDeth
Libraries & Projects>Libraries

IntelliJ Elixir - Elixir plugin for JetBrain's IntelliJ Platform
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
#intellij
289 35421 110
New
hariharasudhan94
Questions & Help>Questions

What is the best IDE for elixir?
I would like to know what is the best IDE for elixir development?
#code-editors
33 49503 23
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
msaraiva
Your Libraries & Projects>Libraries

Surface - A component-based library for Phoenix LiveView
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
/phoenix#liveview#livecomponents#surface
564 42633 214
New
Patoshizzle
Questions & Help>Questions

(Postgrex.Error) FATAL 28P01 (invalid_password) password authentication failed for user “postgres”
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
#ecto#postgres#troubleshooting
10 28696 20
New
siddhant3030
Questions & Help>Questions

How can I write a raw sql query?
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
/phoenix#ecto
13 19061 20
New
joeerl
Questions & Help>Questions

Upgrading Elixir - how to check versions, delete, and upgrade?
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
#upgrade
63 40940 13
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement