mbenatti
Proxy to use a different IP in each request
Hello folks,
I’m thinking to write a service to scrape data from a website, but the first requirement is to use some Proxy to change ip address per request.
I would love to hear about ideas of how to do it using Elixir/Plug/Phoenix.
Most Liked
NobbZ
That sounds concerning - don’t scrape unless you have permission by the site owner. It’s best to ask for a proper API instead.
Shrike
I’ll ignore the permissions issues, and send you to https://luminati.io they can give you basically everything you need except google searches
idi527
Plug and Phoenix are wrappers around http servers, not http clients. And for scraping you probably need an http client. So you can use something like HTTPoison which seems to have some support for proxies.
HTTPoison.post!("localhost", "body", [], proxy: {:socks5, 'localhost', 1080}, socks5_user: "user", socks5_pass: "secret")
Nemon
Good guy Shrike. Indeed, Luminati is cool. However, I found a cheaper alternative with kinda even higher success rates Smartproxy. Check it out in the future.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"