benperiton
Progmatically access a webpage and download a file
Hi,
I have a service that I use that doesn’t provide an API or FTP to get the data that I need, so at the moment it involves logging in, filtering on some dates, then downloading the file.
I’ve seen Chroxy, which looks like it might work using the ChromeRemoteInterface - has anyone used this for something like this? Alternatively, I could write it in JS and use Puppeteer, but how would I then call that from Elixir and process the resulting downloaded file? (It would also mean I would have to have NodeJS installed on the server, which although not a problem, just means another thing to keep updated)
Has anyone had to do this already?
Thanks!
Most Liked
josemrb
I’ve used Hound to drive a Firefox session with Selenium.
You could try to navigate to the download page then grab the cookies and the URL from the generated link and use :httpc to download the file.
# Sample config for Hound
config :hound,
browser: "firefox",
driver: "selenium"
Also there is a ready to use Docker image with a compatible version of Selenium Standalone with Firefox installed.
$ docker run -d -p 4444:4444 --shm-size=2g selenium/standalone-firefox:3.4.0-francium
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers. By developers, for developers.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"