@_russellb: My list:

1. Floki - fetching data from HTML pages
2. HTTPoison - fetching files (also HTML pages)
3. Poison - decoding JSON for example when you have json in element attribute value
4. NimbleCSV - some data you can fetch from CSV files on some pages for example export search results to CSV
5. ExVCR - recording library for HTTPoison - useful in TDD (Test Driver Development)
6. Retry - just to retry
7. Hound - fallback for HTTPoison and Floki for pages using JavaScript frameworks/libraries

There is also Meeseeks (really good replacement for Floki), but for me it requires Rust that in my case I need to compile (Funtoo Linux) that takes lots of time on my old laptop.

Web scraping is simple

I’d say this is a very generalized statement that should be cleared up for future readers bumping into your post. Not being disrespectful, but saying something like that without some real meat could get a new person in trouble.

But trust me - If you find yourself scraping 5-10 million jobs a day, it quickly becomes “not simple”. The premise of crawling/scraping is not complex, but I can assure you that sustaining it for 7-8 years on end and returning data in a timely fashion to paying customers is not easy at all.

Some more tips from my view, having done this for so long (without a single legal issue):

• Try to form a personal relationship with an IP provider - yes you can use the publicly available providers you mention above and there are tons, but none of those will scale to the numbers I needed to hit in any reasonably economic way. Easier said than done, but ask around - exhaust friends in SEO and marketing fields.
• Start slow. If you only have like 100 IPs to work with, don’t touch a target more than 1 time per hour with the same IP to start unless you plan to treat it like a “smash and grab”.
• Link your IPs to user agents somehow. Meaning if you pull IP #1 and go off to hit a site and randomly grab a UA string to roll with it - make sure the next time you show that IP - you come with the same UA string.
• Have some controls for measuring data quality over time. This may need to be manual in your case. You’d be amazed at how many big sites now start throwing you trash data that looks correct at a glance.
• When you think you have a crawl script ironed out - be sure you toss it at a tool like the EFF Panopticlick (and others) to look for obvious fingerprinting you may have not plugged. https://panopticlick.eff.org/
• Make sure, if using a headless browser you’re plugging all the massive, truck-sized holes they all expose… Things like mocking the navigator.platform (and it better match the UA string) are major gaps I see all the time.

I think the choice depends on whether you need to evaluate js or not.

If you do, then in elixir there is hound [0] for this. I haven’t used it though since I usually use selenium with python. Actually, I usually try to avoid evaluating js completely by figuring out what apis it calls and calling them myself.

If you don’t, then just an html parser like floki [1] will do. You might be interested in reading the source code for magnetissimo [2] to see how a scraper works with it.

[0] https://github.com/HashNuke/hound
[1] https://github.com/philss/floki
[2] https://github.com/sergiotapia/magnetissimo

@oltarasenko Couple bits of feedback that are common mistakes I see people making when using Floki.

1. Recommend a safer HTML parser than Floki+mochiweb_html.

   I know it’s nice to not need to start an Elixir article with “and then install Rust”, but web scraping is exactly the situation you do want an HTML5 compliant parser because you don’t know how well formed the HTML will be and mochiweb_html (Floki’s default parser) can incorrectly parse parts of the HTML if it’s malformed (and potentially just drop those parts silently).

   At the very least use the html5ever parser with Floki, though you may have trouble getting it to compile since html5ever_elixir hasn’t been updated for nine months despite an outstanding need to upgrade Rustler so that it works with more recent versions of Erlang/OTP.

   Better yet, use Meeseeks instead of Floki because it will by default provide you an HTML5 compliant parser based on html5ever that does compile on the latest versions of Erlang/OTP.

   Floki’s mochiweb_html parser has a place, mainly in situations where you are dealing with known, well-formed HTML and you don’t need the weight of an HTML5 compliant parser (like when you’re testing your Phoenix endpoints), but people should know the risk they’re taking if they use it for web scraping.

2. Stop parsing each page four times.

   When you run response.body |> Floki.find(...), you’re really running the equivalent of response.body |> Floki.parse() |> Floki.find(...) which means your four Floki.finds are parsing the whole document four times.

   Instead, try parsed_body = Floki.parse(response.body) then parsed_body |> Floki.find(...).

3. Don’t select over the whole document when you don’t need to.

   Three of your selectors are: "article.blog_post h1:first-child", "article.blog_post p.subheading" and "article.blog_post". That means you’re selecting the same article.blog_post three times, then making sub-selections two of those times. Instead, try something like:

   parsed_body = Floki.parse(response.body)
   blog_post = Floki.find(parse_body, "article.blog_post")
   
   title =
     blog_post
     |> Floki.find("h1:first_child")
     |> Floki.text
   
   author = 
     blog_post
     |> Floki.find("p.subheading")
   ...
   

   Doing that means that instead of walking the whole document each time you want to make a sub-selection you just walk the the portion you’re interested in. In this case when there is only one of the thing you’re making sub-selections on it’s probably not a huge difference, but in cases where you’re sub-selecting over a list of items it can add up.

@_russellb - I’ve been using a combo of NightmareJS via ports (using Porcelain). I’ve been scraping at a pretty large scale since last summer with Elixir. There are some more details in some previous threads of mine if you search.

The only downfall with Nightmare or any JS-capable solution is the CPU/memory resources needed. If you are just needing HTML capabilities, HTTPoison (or swap with HTTPotion, raw hackney, etc) works fine in my experience. Unfortunately, I do not have the HTML only luxury as I’m dealing with single page app, JS required systems. As far as the heavy resource load for JS stuff via headless browsing - I have a distributed elixir/erlang solution in place with dedicated scraping nodes I can bring up dynamically and dispatch work to from a master node. Everything is running headless on linux boxes (some in our own datacenter and some on AWS spot instances).

For parsing the results, I use Floki like most people do. Speed is never an issue here for me so I’ve not bothered with the other parsers out there.

FWIW, we came from a lot of legacy Ruby/Mechanize code. The Elixir stuff is much, much easier to maintain and way more stable. I do have times where Nightmare acts up and doesn’t release resources, but I resolve that with some elixir-based cleanup tasks that run every few minutes to check for zombie processes that get missed. Porcelain helps a lot for the port logic, but is not 100%. I also keep all crawling off my main logic node so I can afford to lose a crawl and just bounce a node without sweating it. I’d recommend something similar if you need large scale throughput. I just assume the job was lost if I do not get a reply back from a crawling node within five minutes and send it to another node.

Good luck!