Happy to read people actually using hot code re/loading in the article’s comments on HN.

Me too. But I didn’t have deeper things to say that wasn’t covered by Learn You Some Erlang and the AppSignal guide.

Best part of this post is what it lured out in the hacker news comments. Love when a Discord dev pops up.

Nice to see a bunch of peoole using it.

If you want a deeper look at it Bryan Hunter’s first episode on BEAM Radio “Punking the Servers” and his 2023 talk at GigCity Elixir were both fascinating. Not code-level typically but a fair amount of detail. I think the HCA talk at Code BEAM NYC was a deeper look but it isn’t out yet.

We used hot code reloading in production occasionally at my previous position. We only used it when things were dire, because we were nervous something bad might happen, but it never did. It worked every single time.

It was mainly useful if something went out slightly broken, a missing where clause or a faulty comparison, a team member needed a tweak in validation to finish onboarding a customer, or if we just desperately needed an extra log message. Or maybe the CI job was not working or Kafka was looping on a single malformed event and crashing, causing a backup.

Check out the current tag, modify that module, run a script that would upload that file to each k8s pod and run a Code.compile on it. Worked every time, just takes a couple seconds. Then get code committed and the next deploy would have that code. We never had to modify state.

I use hot code loading a lot in a monorepo with a baseline app and client-specific isolated features.

There are few rules :
Client features depend on the baseline app api with tight but unidirectional coupling so they can be in the same codebase to be easily tested.
The base app can never call client code and has no knowledge of it.
Two client features cannot have knowledge of each other.
Of course client A code has no knowledge of client B code either.

So in dev I can load some, or all client specific features. In testing too, everything stays in sync because it is just a single codebase.

In CI, after the tests, both the client namespace & tests are wiped before the release gets built.

Then the specific features are injected into specific instances and persisted to be reloaded at startup in case of an app restart. There is some plumbing that allows to remove a feature, upgrade it, make it available to the end user, hide it.

What I like with this setup is that I keep it simple but client specific code never makes it into the release. Also, a breaking change in the main app instantly pops up by breaking the unit tests of client features since it’s just a single codebase with parts that happen to be selectively loaded later.

Of course there are a few specifics, like how are specific parts then delivered to a live system, but after reading F. Cesarini’s Erlang books this kind of thing did not seem too weird anymore to me.

I’m at a very small scale and every client runs a separate instance, so what works here for me is not universal. I use hot code loading a bit like a feature flag system built into the runtime, but the features are not idle, they just aren’t there to begin with.

I agree, it’s just that in a world of containers and blue-green deployments this is also not as compelling as it was 10+ years ago. And Phoenix’s team took special care to handle draining of connections on shutdown, so Phoenix seems more or less perfect for blue-green deployments.

Agreed as well, if hot code reloading was somehow simpler I’d give it an honest chance. But when I last researched it 2-3 years ago it just looked way too involved given the alternatives.

I could be missing out and not realizing it, but it’s also not one of the things that seems so crushingly valuable. If it was, we would have very visible blog posts and videos about it. So I made a half-informed decision not to pursue it.