enkr1

Issue of SERVER ALERT: Fatal - No application protocol, SERVER ALERT: Fatal - No application protocol, etc

Hi community,

I had a server running a while ago and it stopped running suddenly. I observed the logs and suspected somebody is trying to brute force my site by accessing different common WordPress (or something else’s) routes:

===== ALIVE Mon Dec  5 18:59:26 UTC 2022

===== Mon Dec  5 19:09:52 UTC 2022
19:09:52.249 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
19:09:52.611 [info] Plug.SSL is redirecting GET /conf/.env to https://mysite.com with status 301
19:09:52.960 [info] Plug.SSL is redirecting GET /wp-content/.env to https://mysite.com with status 301
19:09:53.331 [info] Plug.SSL is redirecting GET /wp-admin/.env to https://mysite.com with status 301
19:09:53.682 [info] Plug.SSL is redirecting GET /library/.env to https://mysite.com with status 301
19:09:54.056 [info] Plug.SSL is redirecting GET /new/.env to https://mysite.com with status 301
19:09:54.424 [info] Plug.SSL is redirecting GET /vendor/.env to https://mysite.com with status 301
19:09:54.777 [info] Plug.SSL is redirecting GET /old/.env to https://mysite.com with status 301
19:09:55.160 [info] Plug.SSL is redirecting GET /local/.env to https://mysite.com with status 301
19:09:55.540 [info] Plug.SSL is redirecting GET /api/.env to https://mysite.com with status 301
19:09:55.874 [info] Plug.SSL is redirecting GET /blog/.env to https://mysite.com with status 301
19:09:56.242 [info] Plug.SSL is redirecting GET /crm/.env to https://mysite.com with status 301
19:09:56.609 [info] Plug.SSL is redirecting GET /admin/.env to https://mysite.com with status 301
19:09:56.962 [info] Plug.SSL is redirecting GET /laravel/.env to https://mysite.com with status 301
19:09:57.311 [info] Plug.SSL is redirecting GET /app/.env to https://mysite.com with status 301
19:09:57.648 [info] Plug.SSL is redirecting GET /app/config/.env to https://mysite.com with status 301
19:09:58.003 [info] Plug.SSL is redirecting GET /apps/.env to https://mysite.com with status 301
19:09:58.381 [info] Plug.SSL is redirecting GET /audio/.env to https://mysite.com with status 301
19:09:58.728 [info] Plug.SSL is redirecting GET /cgi-bin/.env to https://mysite.com with status 301
19:09:59.079 [info] Plug.SSL is redirecting GET /backend/.env to https://mysite.com with status 301
19:09:59.455 [info] Plug.SSL is redirecting GET /src/.env to https://mysite.com with status 301
19:09:59.829 [info] Plug.SSL is redirecting GET /base/.env to https://mysite.com with status 301
19:10:00.186 [info] Plug.SSL is redirecting GET /core/.env to https://mysite.com with status 301
19:10:00.551 [info] Plug.SSL is redirecting GET /vendor/laravel/.env to https://mysite.com with status 301
19:10:00.891 [info] Plug.SSL is redirecting GET /storage/.env to https://mysite.com with status 301
19:10:01.234 [info] Plug.SSL is redirecting GET /protected/.env to https://mysite.com with status 301
19:10:01.562 [info] Plug.SSL is redirecting GET /newsite/.env to https://mysite.com with status 301
19:10:01.904 [info] Plug.SSL is redirecting GET /www/.env to https://mysite.com with status 301
19:10:02.263 [info] Plug.SSL is redirecting GET /sites/all/libraries/mailchimp/.env to https://mysite.com with status 301
19:10:02.607 [info] Plug.SSL is redirecting GET /database/.env to https://mysite.com with status 301
19:10:02.952 [info] Plug.SSL is redirecting GET /public/.env to https://mysite.com with status 301
19:10:03.352 [info] Plug.SSL is redirecting GET /ec2-18-142-114-116.ap-southeast-1.compute.amazonaws.com/.env to https://mysite.com with status 301
19:10:03.701 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:10:04.047 [info] Plug.SSL is redirecting GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php to https://mysite.com with status 301
19:11:01.689 [notice] TLS :server: In state :hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
 - :no_suitable_ciphers
19:11:30.675 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
19:11:31.519 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:14:56.662 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301
19:14:56.725 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301
19:14:58.158 request_id=Fy37nzyNa_Qcb7QAASix [info] GET /
19:14:58.159 request_id=Fy37nzyNa_Qcb7QAASix [info] Sent 200 in 952µs
19:14:59.746 request_id=Fy37n5svCSUZSNUAASkB [info] GET /
19:14:59.746 request_id=Fy37n5svCSUZSNUAASkB [info] Sent 200 in 496µs

===== ALIVE Mon Dec  5 19:29:59 UTC 2022

===== Mon Dec  5 19:42:04 UTC 2022
19:42:04.508 [info] CONNECTED TO Phoenix.LiveView.Socket in 36µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "LD8vAQUbQgAJJCkbXSBYLQx7W1s6QW58oyBDKoqo1WJphU7Uf3bowvCQ", "_mounts" => "11", "_timezone" => "Asia/Singapore", "_track_static" => %{"0" => "https://mysite.com/assets/app-6676b9997926b9d99094855080ac6f52.css?vsn=d", "1" => "https://mysite.com/assets/app-f7aa12f5be2e70bd89dc872c8be91fc8.js?vsn=d"}, "vsn" => "2.0.0"}
19:45:43.028 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
19:45:43.550 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:45:44.615 request_id=Fy39TSXxcXD4KfIAASmR [info] GET /.env
19:45:44.615 request_id=Fy39TSXxcXD4KfIAASmR [info] Sent 404 in 293µs
19:45:45.754 request_id=Fy39TWnTBiF_6BMAASnB [info] POST /
19:45:45.754 request_id=Fy39TWnTBiF_6BMAASnB [info] Sent 404 in 312µs

===== Mon Dec  5 19:56:59 UTC 2022
19:56:59.333 [info] Plug.SSL is redirecting POST / to https://mysite.com with status 307
19:56:59.814 [info] Plug.SSL is redirecting GET /.env to https://mysite.com with status 301
20:01:54.022 [info] Plug.SSL is redirecting GET / to https://mysite.com with status 301
20:01:54.752 request_id=Fy3-Lwaf8LFiPlIAASnx [info] GET /
20:01:54.752 request_id=Fy3-Lwaf8LFiPlIAASnx [info] Sent 200 in 506µs
20:01:55.574 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

20:01:55.900 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

20:01:56.238 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

20:01:56.571 [notice] TLS :server: In state :hello at tls_record.erl:564 generated SERVER ALERT: Fatal - Unexpected Message
 - {:unsupported_record_type, 128}
20:01:58.052 [notice] TLS :server: In state :hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
 - :no_suitable_ciphers
20:01:58.881 [notice] TLS :server: In state :hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
 - :no_suitable_ciphers

I am here to ask for help on why a GET & POST would trigger such errors and why would it break the whole site (stopped completely).

Thank you so much in advance!

Best wishes,
Jing Hui PANG

13 comments

#logs #tls #plugssl

13 2203 13

2022-12-19 08:49:12 UTC

Marked As Solved

milangupta

For what it is worth …

Locking down the server is critical … the number of bots out there scanning and attempting brute-force attacks is astounding !!
There were a couple of key things I did to get my application to stay up :

front end firewall (only 443 exposed)
ufw on ubuntu (again only 443 exposed)
proper ssl cert installation - a good site to verify install - SSL Server Test (Powered by Qualys SSL Labs)
review your router.ex logic to ensure you are locked down (incl. serving static assets etc.)
use eg. paraxial.io to get the ability to review/block traffic via plugs (even though it is a bit too late, still will give you a control).
update to latest and greatest on ciphers / tls etc. My config if useful -


config :yourapp, yourappWeb.Endpoint,
  url: [host: "yoursite", port: 443],
  https: [
   port: 443,
   cipher_suite: :strong,
   otp_app: :meta,
   keyfile: "*** certificate files ***",
   certfile: "*** certificate files ***",
   cacertfile: "*** certificate files ***",
   honor_cipher_order: true,
   ciphers: [
    'TLS_AES_128_GCM_SHA256',
    'TLS_AES_256_GCM_SHA384',
    'TLS_CHACHA20_POLY1305_SHA256',
    'ECDHE-ECDSA-AES128-GCM-SHA256',
    'ECDHE-RSA-AES128-GCM-SHA256',
    'ECDHE-ECDSA-AES256-GCM-SHA384',
    'ECDHE-RSA-AES256-GCM-SHA384',
    'ECDHE-ECDSA-CHACHA20-POLY1305',
    'ECDHE-RSA-CHACHA20-POLY1305',
    'DHE-RSA-AES128-GCM-SHA256',
    'DHE-RSA-AES256-GCM-SHA384'
    ],
   eccs: [
    :x25519,
    :secp256r1,
    :secp384r1
    ],
   secure_renegotiate: true,
   reuse_sessions: true,
   versions: [:"tlsv1.3", :"tlsv1.2"]
  ],
  transport_options: [socket_opts: [:inet6]],
  force_ssl: [hsts: true],
  #force_ssl: [rewrite_on: [:x_forwarded_proto]],
  cache_static_manifest: "priv/static/cache_manifest.json",
  server: true,```

Post #2

Also Liked

milangupta

This is basically saying something is trying to connect to your ssl port using a protocol that isn’t configured. This isn’t a bad thing (in my humble opinion) as you should, ideally, be restricting/supporting only the latest TLS/Cipher suites.

Here is a good article I found … https://www.baeldung.com/java-ssl-handshake-failures

Post #4

BrightEyesDavid

If you don’t already have a reverse proxy/webserver between the internet and your app, you could try running Caddy, or Cloudflare, or both, in the hope that, when configured appropriately, they would prevent whatever request is causing your app trouble from getting to the app.

Caddy is very easy to set up and automatically handles SSL certificates.

Post #6

milangupta

There are still too many variables here and the scope of where the actual issue is, is very broad. I haven’t seen crashes like you are seeing (am running the latest elixir/phoenix stack on ubuntu).

My instincts are pointing me to the redirects … eg. 301 is a permanent redirect … but what is https://mysite.com ? This looks like a “default” that hasn’t been configured. You will need to go deep into SSL & DNS (as well as the Phoenix stack, i.e. configuration of your endpoint and router) to get to the bottom.

My recommendation is to keep simplifying your config until you stop seeing the issue. If you get really lucky, you will be able to recreate the crash on demand by injecting a http request to your endpoint. That would exponentially increase your chances of tracing/catching this gremlin.

Post #13

enkr1

Hi @milangupta,

Thank you for the detailed information!

I have deployed the new configuration on my prod.exs and I can still observe something like this in my logs:

===== Thu Dec  8 03:03:55 UTC 2022
03:03:55.967 [notice] TLS :server: In state :hello at ssl_handshake.erl:3435 generated SERVER ALERT: Fatal - No application protocol

03:03:56.405 [notice] TLS :server: In state :hello at ssl_handshake.erl:3435 generated SERVER ALERT: Fatal - No application protocol

03:03:56.830 [notice] TLS :server: In state :hello at tls_handshake.erl:364 generated SERVER ALERT: Fatal - Protocol Version

Post #3

milangupta

Thanks for sharing. I’ll try this too. I wish there were a better “master” reference and one tested way of configuring this.

One question - why are you using both http on port 4000 and https on port 4040 ? I am assuming you have something else redirecting the normal http/https traffic from port 80 & 443 to these two ports … are you sure the traffic is not due to some redirection loop problem and is actually from external IPs ?

Just feels strange as a ddos attack should be quite a significant volume for (am assuming) cowboy to go down … you must be doing something interesting to attract that kind of attention