I figured I should update everyone here, but I accepted an offer to join DockYard to fill the position they opened up for this. I’ll be formally starting November 3rd, and once I do I’ll be jumping into this head first.

I’ll be posting more in the forum once I have some more concrete steps fleshed out, partly to keep everyone up to date, but also as a means of gathering feedback on what’s planned. More to come later!

I think it’s important to understand that deployment tooling has to integrate neatly into a very wide variety of environments and use cases, so I don’t agree that a nifty UI based tool is the primary solution (although having such a UI which consumes the underlying tools as a library might be really cool for those preferring that kind of thing). On the easy side, you have deploying to a VM/VPC, AWS/Azure/etc., where the tooling is simple and can be very flexible. Things are still fairly easy, but a little more constrained with deploying to services like Heroku, et al. Then you have people deploying releases via RPM or other package formats, perhaps installing in a VM and creating images for deployment to customer sites. Another group is deploying in an environment running Kubernetes, Mesos/Marathon, or OpenShift for example, and there are a lot of variables. How you deploy depends on whether you do blue/green, A/B deploys, or neither and just push the latest into service immediately. Luckily, Nerves has more or less solved the embedded side of things, so we don’t have to worry about that all that much, but the underlying tools still need to support Nerves. Right now that is Distillery, but if it became another tool, that tool would need to support other tools using some of the lower-level primitives to compose their own solution.

My personal take on this is that there is considerable room for improvement on the build side - making it extremely simple to build a deployment package for another platform, regardless of where you are building from. This should be exposed as one of the “primitives”, since some people will want to take that package and do their own thing with it. I think the build tooling should support generating container images, RPMs, etc., either through plugins or out of the box, making it trivial to choose the format you want to deploy with. On the deployment side, I suspect that those running container orchestration platforms like Kubernetes will use the tooling those systems provide to actually handle deployment, but having a flexible build tool will make it easy to integrate Elixir into the environment. In my own experience, working with Elixir under Kubernetes/OpenShift is already very easy, so there is not much else required here. For those deploying to VMs/VPC or AWS/Azure, this is where better tooling is most needed, and where most tooling currently sits. The biggest issue is that one needs a build machine which matches the target machine, more or less exactly. I suspect what Brian meant in his video is that Nerves allows you to build packages for a variety of hardware all from whatever machine you are currently using, via buildroot and a cross-compilation toolchain. If we could do something similar for general Elixir deployment, it would be a huge improvement over the situation today, as it would alleviate the vast majority of issues newcomers experience. Once you get beyond building a release and getting it to a target, the rest is actually quite pleasant with Elixir. There are still some rough patches with regards to configuration and differences between Mix and releases, but those gaps are closing, slowly but surely. I suspect being able to focus on the problem full-time, these too can be solved more elegantly.

Nice update about deploy tools! Thanks @bitwalker

Sounds like a big step in the right direction.

Thanks for the feedback everyone! I’m glad you all like the direction

More or less, to be clearer, the plan is to have Elixir’s configuration file be evaluated and pushed into the application env (i.e. which is exposed by Application.get_env/3) right after Elixir is loaded and started, and before any other applications are started. This mirrors how Mix handles things today, and would allow you to access the application env in your own application right away without having to do any of your own fetching/transformation.

Of course, that’s only true for things which use the config file, for those use cases where you need to fetch configuration from external system, say etcd, you’d still have to follow more or less the existing approach to that scenario. That said, I’m hoping to provide an extension point which allows one to extend the configuration system with custom providers, so that by the time applications start booting, all the configuration is done. I don’t know what that will look like yet, but I think it’s an important part of consolidating configuration into a single approach.

I know José is planning to move what is currently Mix.Config out of Mix and into Elixir as Application.Config, and while initially it should just mirror how Mix.Config works today, I suspect that would provide us a foundation for extension like I mentioned above.

I’m in agreement with @DianaOlympos, @talentdeficit, @wmnnd, and @bitwalker here. Creating a new “deployment tool” specific to the Elixir ecosystem seems like it’s a solution in need of a problem.

When I first started using Elixir, I often heard that deployment was an issue, and I started repeating that as if it was a mantra. The company I worked with at the time started by deploying to Heroku, and that worked well for our purposes. I moved companies, and my current company was also deploying Elixir to Heroku. As our needs grew, I learned how to build releases with Distillery. I put together a deployment pipeline using non-Elixir tools and started shipping releases from our build servers through to staging and production.

And now I question what this “deployment is an issue” comment really means.

Right now my team uses AWS CodeDeploy for deployment purposes. A release artifact (built using Distillery) is bundled with rules for CodeDeploy’s engine. The bundle is pushed to S3 from our build server and registered with CodeDeploy. Our current setup requires manual triggering of a deploy, and we currently use the CodeDeploy UI for that. We utilize the same bundle for both staging and production. Our system loads as much of its configuration as possible at runtime via environment variables.

I don’t solve deployment using Elixir focused tooling. I use generic tooling because it’s a generic problem. I could use the same tools now with Go, Java, C#, etc.

I could also build it into an AMI using Packer, seal it into a Docker image, scp it to a fleet of servers using Ansible, bundle it into a .deb…there’s a list of generic tools that solve these issues.

What I think we need is more documentation around best practices for building releases, managing runtime configuration, integrating with existing infrastructure tooling, and incorporating operating concerns into our codebases and executables. That’s what I’m picking up from this conversation and other conversations I’ve had in the community.

From what I can tell, people want/need:

• A clearer picture of why to use releases and why running in production via mix is not best practice
• Guidance on runtime configuration management (and particularly helping package authors write configuration-system agnostic packages)
• More information about taking a release and running it on Ubuntu/Amazon Linux/Docker/K8s/the back of a turtle; this includes how to publish the release to the unit, write startup scripts, redirect logs to files or services, manage starting/stopping the release
• Expanded guides on building operations logic into the codebase and releases (e.g., performing Ecto migrations without mix, and other things one normally would have made a rake task or the like for)

If someone asks you, “Now that I’ve built it, how do I launch it?” they’re looking for instructions. That doesn’t mean another tool is necessary.