You need to follow redirects then, you can do that by passing the option follow_redirect: true, max_redirect: 20 for whatever values you want. Or follow them manually.

I’m not sure why there would be a point to use a headless browser, should just use HTTPoison.

My only suggestion is that as long as you’re using a headless browser you’ll never know what is actually happening. You need to download the document in a regular browser, then look at the network activity in your browser’s dev tools to see what actually happened in terms of redirects etc, then use an HTTP client lib (HTTPoison is the usual choice) to follow the same path.

It’s perfectly possibly that the site you’re hitting uses a combination of redirects AND cookies. Advertising & tracking sometimes complicate the heck out of things. I’ve seen a single page load result in a series of 8 redirects each of which added some damned tracking/advertising info until the request was finally answered.

Uh… that’s a server bug, like big-time server bug… o.O

Well HTTPoison is the thing for that, plus processing the HTML through Meeseeks or something if you need to do that.

Thank you for your response! When I tried using follow_redirect: true, I found myself stuck in a loop where I was being redirected to the same URL. The reason I’m trying to do this in a headless browser is because doing this manually is time consuming and it would be better to have this run over night without any need for user input. I was thinking that if there was an Elixir library that allows you to inspect the HTTP traffic then I could identify the file that I need.

See that’s exactly what I was thinking as far as the server bug goes. My guess is that it was possibly set up that way to prevent web scraping and if that’s the case then there really isn’t much that can be done. I will definitely look into meeseeks, thanks for the suggestion!