michaelterryio
Plug/Cowboy "connection closed" fixed with Nginx proxy server
We have a service with a Plug interface. It calls a vendor API with requests and then later receives callbacks with payloads responding to the requests.
A couple weeks ago, the vendor’s error page started reporting that it was getting “connection closed” when hitting our callback endpoint about 1% of the time. This was odd to us because nothing had changed. We hadn’t made a commit on this service in over a year and it’s running on the same EC2 instance it always has.
The fix was to put an Nginx proxy server in front of the Elixir service.
While it’s nice to have a fix, why it happened is still a mystery. What would have been a good way to debug this? Is it more likely a Plug or Cowboy issue? My best guess is that the vendor changed or upgraded their HTTP client and there’s an incompatibility with Cowboy. Their support did not suggest this as a possibility when we talked to them, though. They felt there was something wrong with our application–we talked to them before the proxy fix, however. Any other thoughts?
Most Liked
jakemorrison
That sounds more like network problems than an application incompatibility. Is there some more specific error code that they are getting? Or just an HTTP timeout?
You can log the whole conn and get details about their HTTP request.
Using Nginx adds a “buffer” as it will queue requests for a while waiting for the back end to respond. That can end up being a problem on high volume applications, and things get more reliable when we remove Nginx. https://www.cogini.com/blog/serving-your-phoenix-app-with-nginx/
Cowboy is generally a pretty compatible HTTP stack, though sometimes you need to set protocol options to handle things that Nginx was handling for you. e.g.
config :foo, FooWeb.Endpoint,
http: [
compress: true,
protocol_options: [max_keepalive: 5_000_000]
],
jakemorrison
I guess SSL/TLS version or parameters. One of the machines in their pool is requiring something stronger than what Cowboy is configured for, or vice versa.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers. By developers, for developers.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"