JoeZMar

JoeZMar

Multiple Hound Sessions, but with different IP addresses?

I am creating a web scraper/bot. Because the site that I am scraping is very JS heavy I chose Hound with PhantomJS running.

Right now if I have 5 people that want to use the bot, they log in and provide the username and password for their account for the site they want to scrape. Each account spawns a new process, and uses Hound to bot the site.

My question is, is there any way to spawn each process with a different IP address? I think that you can configure the PhantomJS to run off a proxy, but is there a way to have each browser run off a proxy?

Most Liked

Shrike

Shrike

From my experience with watir it should be possible, I’m running 84 different phantomJS processes on one computer without any cross contamination between them, and that’s ruby I wonder how easy it should be with Elixir.
Only reason it’s 84 is that on Amazon Linux the limit is 85 before I finished all the ports, and the memory was low any way so didn’t try to fix it or even check what was the problem.

Shrike

Shrike

My problem was with phantomjs that after the 85 instance it couldn’t get a new port so it wouldn’t start, you should test it before moving on, my scalability was limited by that + memory phantomjs just love memory even for simple sites so 4gd server couldn’t hold more.
I used Sinatra to move the requests to multiple servers keeping the state in json and passing it back to our elb on fails, each server could deal with any request.
But to say the truth I got this idea from here and wouldn’t have used it if I didn’t know Elixir, but I had to get everything up and running in about a month if I had 3 months I would have done it using Elixir.
In watir proxy definitions are defined for each phantomjs by the use of the driver so I’m guessing you should find it in the same place in hound.
Also if you need loads of ips using proxies you should check luminati.io their data center offer is not bad.

Shrike

Shrike

Never tried to run more than one session per instance, just thinking about all the concurrency problems that might happen would get my pants dirty, especially when running so many instances.
Come to think about it I already have 1k concurrent phantomjs sessions (12 computers) and I tested it and everything is working great even in prod (AWS t2.medium).
I would test chrome + Firefox headless before making a decision, too many unknown variables for me to give any meaningful advice.
Do some testing understand your constraints memory/available instances for each platform etc.
What I did is restart each phantomjs instance after I used it to make sure my memory is clean, you might not need to do it.
I would also recommend opening your browser in advance, when I tried opening each when needed I lost 10 seconds in overall time, it took the server 1sec to open each session and that wasn’t truly concurrent.

PS- not sure what would happen with session instead of instances as I never tried it always had to either deal with concurrency or one instance only and always with ruby, which is not the best case scenario, I wonder how hound would deal with it, and if each session uses a different port too.

Where Next?

Popular in Questions Top

lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New

Other popular topics Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement