pillaiindu
Where do I put driver configuration for hound in a (non Phoenix) Elixir project for web scraping?
Hi Everyone!
I want to do some web scraping, for which I wanna use the hound package as it have everything required for web scraping.
I know how to use it in a Phoenix project for testing, which I learnt from Phoenix inside out, thanks to @shankardevy. In a Phoenix project we had to put config :hound, driver: "phantomjs" in config/config.exs, if we were using phantomjs.
But here the situation is a bit different.
- Its not a Phoenix project.
- I want to use it in a file inside
lib/nottest/, as this time I’m using it for web-scraping instead of testing.
Thank you!
Marked As Solved
DevotionGeo
I don’t know if it’s from the start or it changed in the recent versions (I predominantly created Phoenix apps, not bare Elixir apps), but a mix new something won’t add the config.exs by default.
Add a config.exs under a config\ folder with the following code. If you’re using something other than phantomjs edit the code accordingly.
use Mix.Config
config :hound, driver: "phantomjs"
Also let us know if it worked or not.
Also Liked
NobbZ
Of course :hound will fallback then to the defaults.
If you do not want to use the defaults, you need to configure them.
Though I’d check if there is some API for hound that allows for runtime configuration of single scrapers rather than a global boot time configuration…
Phoenix is just a library. It doesn’t make your projects different in any way, they are just plain OTP applications.
NobbZ
Seems like you can use Hound.Helpers.Session.start_session/1 to start a runtime configured session.
PS: I really don’t like hounds API, as it has a lot of hidden magic, I prefer wallabys explicit passing around of the responsible client/connection.
NobbZ
It should just work…
Make sure its not runtime: false and not only: :test in your deps.
NobbZ
Looks like :hound hasn’t been started.
How do you start iex and how does your application/1 in mix.exs look like?
wolfiton
You don’t have a webdriver running for example phantomjs --wd
This error here is what i posted above in case you want a ref
** (RuntimeError) could not create a new session: econnrefused, check webdriver is running
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"