Thought a picture might help (and could be interesting for readers ):

071201×1057 93.6 KB

The yellow box is where the nightly quantum job happens. The label underneath each graph is the point of deployment of the new build without the ERL_COMPILER_OPTIONS='native' env var set. As you can see, since this deploy the load-arg has been lower than previous. Thoughput is down, but it tends to roll in waves, so another 5hrs should see it peak again. I’m looking forward to seeing whether there’s another memory incident with the flag disabled.

Unfortunately there are no magic env vars like that. From your description, though, it seems to me like maybe you have an endpoint that is very memory-hungry. My usual suspects to look for would be loading a lot of data from the database with something like Repo.all on a huge table.

there is the fullsweep_after or ERL_FULLSWEEP_AFTER http://erlang.org/doc/man/erlang.html#system_flag-2 (can’t deep link for some reason…)

But I would be very careful using that, to the level of warning against using that… it’s treating the symptoms rather than the problem… (and can create a whole lot of other problems… increased cpu/latency - unpredictable latency etc etc)

what is the nature of your system? do you have periodic cron/quantum jobs that does the spike… is it one user that have obscene amount of data etc etc.

so identify what process/code is responsible for the spike…

also what OTP/elixir is this… upgrading could help a bit here and there…

do you have periodic cron/quantum jobs that does the spike

No, it seems to be truly random. I do have a nightly quantum job that migrates Redis metrics to postgresql for persistence, but the spikes aren’t around that time. I can go for days without issue, then I’ll experience an outage and a 3-4x memory spike. Bounce the server and it’ll run fine again… till the next time.

Traffic-wise, there’s no peak at the time of the spike; requests are generally quite consistent.

is it one user that have obscene amount of data

Yeah, good thought. I should’ve mentioned in the original post: this is a true “micro-service”, so there’s no auth, and very little processing, mostly reading binary data and returning it as json.

there is the fullsweep_after or ERL_FULLSWEEP_AFTER

Actually, this is very interesting. From the docs (I’ve bolded the parts relevant to my case):

A few cases when it can be useful to change fullsweep_after:

• If binaries that are no longer used are to be thrown away as soon as possible. (Set Number to zero.)
• A process that mostly have short-lived data is fullsweeped seldom or never, that is, the old heap contains mostly garbage. To ensure a fullsweep occasionally, set Number to a suitable value, such as 10 or 20.
• In embedded systems with a limited amount of RAM and no virtual memory, you might want to preserve memory by setting Number to zero.

I think removing ERL_COMPILER_OPTIONS='native' may be the resolution, but if not I’ll try setting ERL_FULLSWEEP_AFTER=0 and see how I get on, and report back on this thread if I do (others may find the results useful).