josevalim
Proposal: moving towards discoverable config files
One of the major differences between running your application as a release and as a Mix project is the differences in configuration. Mix evaluates the configuration right before the application starts, releases evaluates the configuration when your application is compiled.
This implies in a large mismatch of how those two environments are used. For releases, environment variables (read by System.get_env/1) need to be set when the application is compiled and such information may not be available at this point.
Ideally, we would want a release to evaluate the configurations files in config when the release starts. One approach would be to copy the configuration files as is to the release but that’s hard to achieve in practice for two reasons:
-
A config file may import other config files and often importing those files happen dynamically. For example:
import_config "#{Mix.env()}.exs". The dynamic import makes it hard for release tools to know which configuration files must be copied to a release, especially in cases like umbrella projects, where a developer may load configuration across projects -
Even we copy today’s configuration files to a release, those configuration files rely on
Mix, which is a build tool and therefore it is not available during releases
To solve those issues, we need to make sure we can discover all imports of a configuration file without evaluating its contents. We also need to introduce a new module for configuration that does not depend on Mix.
This is the goal of this proposal.
Application.Config
This proposal is about introducing a module named Application.Config. It will work similarly to the existing Mix.Config, except it belongs to the :elixir application instead of :mix. This allows releases to leverage configuration without depending on Mix.
The user API of Application.Config is quite similar to Mix.Config. There is config/2 and config/3 to define configurations. There still is import_config/1 to import new configuration files with one important difference: the argument to import_config/1 must be a literal string. So interpolation, variables or any other dynamic pattern is no longer allowed.
In order to help with configuration management, we will introduce a project option in your mix.exs, named :config_paths to help manage multiple required and optional configuration files.
In the next section we will provide an example of how configuration files used by projects like Nerves and Phoenix will have to be rewritten and then we will discuss how integration with release tools such as distillery will work.
A common example
Projects like Nerves and Phoenix generate files with built-in multi-environment configuration. Today, this configuration has an entry point config/config.exs file that imports an environment specific configuration at the bottom:
# config/config.exs
use Mix.Config
config :my_app, :some_shared_configuration, ...
import_config "#{Mix.env()}.exs"
And then each config/{dev,test,prod}.exs provides environment specific configuration. For instance:
# config/dev.exs
use Mix.Config
config :my_app, :some_dev_configuration, ...
The issue in the example above is the use of dynamic imports, such as import_config "#{Mix.env()}.exs". We will address this by defining both config/config.exs and config/#{Mix.env()}.exs as configuration entry points in your mix.exs:
# mix.exs
def project do
[
...,
config_paths: ~w(config/config.exs config/#{Mix.env()}.exs),
...
]
end
And now we can define those configuration files without dynamic imports:
# config/config.exs
import Application.Config
config :my_app, :some_shared_configuration, ...
# config/dev.exs
import Application.Config
config :my_app, :some_dev_configuration, ...
In Phoenix, the config/prod.exs case may link to a separate prod.secret.exs file. While we could also refer to this file in the :config_paths configuration in the mix.exs file, because it is only specific to production, it is more straight-forward to continue importing it at the bottom. So a config/prod.exs would look like this:
# config/prod.exs
import Application.Config
config :my_app, :some_prod_configuration, ...
import_config "prod.secret.exs"
By adding :config_paths, we are able to move the dynamic configuration to the mix.exs file and make the order that configuration files are loaded clearer.
A FarmBot example
Nerves projects tend to rely extensively on configuration files. So let’s look into existing open source Nerves projects and see how this proposal will fare. Let’s take a look at FarmBot v6.4.1.
The questions we want to answer are: if we move the FarmBot project to the proposed Application.Config, will they be able to express of all the existing idioms they do today? And, even further, will their configuration files become simpler or more complex?
From looking at its config/config.exs, we can already see a pattern that won’t work in releases: the use of Mix.env and Mix.Project.config.
We can see those variables are used to dynamically import configuration, which Application.Config won’t allow.
Those idioms are perfectly fine with how configurations work in Mix today. But they will no longer with a release built on top of Application.Config.
The solution is to move all of those imports to the :config_paths option in mix.exs. However, note that some of those dynamic imports are optional, so we will also need the ability to explicitly tag them as such:
# farmbot/mix.exs
def project do
[
...,
config_path: ~w(config/config.exs config/#{Mix.env()}.exs) ++
optional_config_paths(@target, Mix.env())
...
]
end
defp optional_config_paths("host", env),
do: [{:optional, "config/host/#{env}.exs"}]
defp optional_config_paths(target, env),
do: [{:optional, "config/target/#{env}.exs"}, {:optional, "config/target/#{target}.exs"}]
We believe this approach is an improvement to the previous one because it allows all environment and target specific handling to remain in the mix.exs file and not scattered around multiple configuration files.
Using it in releases
In the previous sections, we have outlined Application.Config which no longer depends on Mix and has a restricted import_config.
Now that we are able to see all of the configuration files that affect our system, a release tool, such as distillery, should be able to traverse all of those configuration files and merge them into a final config/release.exs that will be part of your release. In fact, Elixir will provide a convenient API that performs such operation, streamlining the release assembling process.
Unresolved topics
There are two important topics that we have not included in this proposal and they will be discussed in a further step.
-
What about umbrella projects? Umbrella projects also rely on configuration and we need to make sure the listed mechanisms also work well with umbrellas.
-
How to avoid common pitfalls? Even though we will migrate to
Application.Config, there is nothing stopping a developer from accessing Mix (and the module defined in themix.exsfile) from their new config files. As we have seen, this may lead to errors when running releases, as releases do not have Mix available. To address this, we may introduce checks when assembling releases that make sureMixis not invoked in configuration files, raising appropriate error messages in case they do.
Summing up
We propose a new Application.Config module and a new :config_paths project option that allows release tools to discover all of the relevant configurations in a system. Release tools can then merge and copy those configuration into releases and execute them as part of the release process, allowing dynamic calls such as System.get_env/1 to work in development and in production transparently, with or without releases.
Most Liked
josevalim
Thanks everyone for your feedback! 
We have learned a couple things from this discussion:
-
It is probably simpler to tackle the introduction of
Application.Configas a separate proposal asApplication.Confighas in itself the goal of decoupling configuration Mix, regardless of features that may be added or remove from the configuration API in the process -
The proposal, as originally written, is not enough. Most agree that we need a more explicit mechanism to separate runtime/compile-time configuration. Although we do not necessarily agree on how hard this separation has to be or its API
There are also more operational concerns regarding the semantics of a chosen solution. What happens if the on_boot callback fails? Can we do something about it? This is important on Nerves devices as you usually don’t want failing to boot to bring the VM down.
For now, I have decided to hold on moving this proposal forward. The reason is that, before we have minimal releases in Elixir itself, we won’t be sure on how to answer some of those questions. It may also be that, by adding releases to Elixir itself may solve some of those problems. For example, what if, once we add releases to Elixir, it becomes impossible to run MIX_ENV=prod mix run by default and instead it points you to a release? If this happens, the distinction between Mix and Releases is completely gone, at least for the prod environment. We are speculating here but the point still stands: without releases in Elixir, we cannot be 100% sure the direction we choose is the best way to go. So we will proceed on adding releases to Elixir and then hopefully this discussion can be brought back to life.
Once again, thanks for all the feedback. It was a fantastic discussion, as always.
Qqwy
I finally was able to find the time to read through this whole thread. The one thing everyone agrees on, is that the current way configuration happens could be improved. The exact opinions people then have about how to improve it and what to improve on exactly are then widely varying; I guess we might still for some of the proposed issues only be looking at a symptom rather than the underlying cause.
Of course, the proposal at hand is only about one single thing (there not being a real way to do currently writing runtime configuration for releases), so let me focus on that one now in more detail. The other issues that have been discussed so far (what to (not) put in a configuration file, other ways to configure an application, how applications ought to read configuration settings that the user put in, and some others) are I think very important, but definitely fall outside of this thread’s intended subject matter.
It is an unmistakable fact that an Elixir project is first compiled, and then run. In some cases, the environment that the application is compiled on is different from the environment where the application is run. This is true for at least releases and Nerves projects. In all cases (not only in these two), both of these steps (compilation and booting+running) happen. Even when we run a project in a Mix project, it is first compiled and only then executed. However, since we usually happen to still be in the same folder with the same files (+environment) available, we can accidentally (e.g. without being conscious about it) depend in our run-time on things we decided during compile-time: re-use our configuration.
So currently, Mix configuration happens in both places in Mix projects because it works ‘by happy circumstance’ also during application startup, but it does not in the other cases (actually, I think this is regardless of the fact of having Mix still available to us at runtime or not, because we cannot depend on the locations of files or other things anymore when the runtime environment(/machine) is different from the compilation environment(/machine)).
To me, it thus feels like the current behaviour is implicit, and it can be made more explicit by making it clear that the difference between compile-time configuration and run-time configuration exists (and has always existed!) and needs to be kept in mind.
The two proposals we currently have, Application.Config and on_boot both only address this problem partially:
-
Application.Configsolves the problem ofMixnot always being available, but hides the fact that compile-time and run-time configuration might be different (or, maybe clearer phrasing: that some beam-applications might read parts of the configuration during compile-time and some others during boot-up or runtime). -
on_bootmakes it more clear that certain pieces of configuration will only be executed during boot-up/runtime, but to be honest it does not feel very clean to me because of the following fact: It seems to indicate that something different needs to happen during boot-up/runtime rather than the compile-time configuration outside of it. it is the word different that I have issue with here.
It feels to me that the ‘default’ of an application would be to read the configuration settings during boot-up/runtime, and only if the application requires special behaviour to happen during compilation-time (because of time- or space-optimizations), then this is where we would opt-in into a special compile-time configuration block, rather than compile-time being the default and opting-in into a special run-time configuration block.
It is only a thought however, because I do see that with the way how configurations currently work (Releases can act on config during compile-time but not during boot-up/runtime), that it might be difficult to build it with this order of specialization (compile-time config being a specialization of run-time config) in mind. But I do think that it is important to mention this, because personally, on_compile feels clearer to me than on_boot.
But besides this, in essence I think the problem is not that ‘releases are broken’ but rather that Mix is able to pretend that the difference between compile-time and run-time configuration does not exist. So rather than building a ‘fix’ for releases, I think that improving on Mix’ behaviour to make it more clear that there is (and always has been) a difference is the way to go.
josevalim
Thanks @hubertlepicki for the follow-up.
One of the conclusions of this conversation was precisely “Most agree that we need a more explicit mechanism to separate runtime/compile-time configuration”. So I believe the direction Distillery 2.0 took of providing an explicit runtime configuration for releases is very much inline with the thoughts of the core team and of the community.
Excellent work from @bitwalker!
sasajuric
I’m generally in favour of the proposed changes because I think they will simplify some frequent scenarios, most notably fetching settings from OS_ENV.
Another benefit is that config is now present on the server in Elixir syntax, not the Erlang one. Though, in my limited experience I can’t recall a single time when I edited sys.config in prod.
I’ve recently posted a rant on app config which was partly motivated by some discussions I’ve had with @josevalim and @bitwalker, partly by some discussions I’ve had with colleagues, and partly by my continuous distaste for ad-hoc config files (which dates back from my rails days).
It’s a long article, but TL;DR is that while I agree that config scripts are very convenient, and even in the current form flexible, the outcome is that app env tends to be bloated with all sorts of things which are not configuration at all. Even in this thread there have been a couple of examples:
config :code_stats,
commit_hash: System.cmd("git", ["rev-parse", "--verify", "--short", "HEAD"]) |> elem(0),
version: Mix.Project.config()[:version]
These are not system configuration parameters, they’re constants derived from the current state of code on the build machine.
config :ui, UI.Endpoint,
url: [host: "localhost"],
render_errors: [view: UI.ErrorView, accepts: ~w(html json)],
pubsub: [name: UI.PubSub, adapter: Phoenix.PubSub.PG2]
As I mentioned in the article, :render_errors and :pubsub are parameters to the library, not configuration.
It could be argued that host is a system configuration, and having it here allows you to change it without needing to redeploy the system. In practice, I don’t think this is a common scenario, and even if I needed to do it, I probably wouldn’t mind redeploying the system. In other words, if you expect that the host will have to be changed and deploying will not be an option, that’s a compelling reason to make host an explicit system property. Otherwise, it’s a case of YAGNI 
Therefore, Instead of further expanding the support for config scripts, say by explicitly separating compile time and runtime scripts, I think that as a community we should first consider how much stuff do we place inside config for mere convenience, and as a consequence how much our config scripts and app envs become bloated with data which is not a part of the system configuration. Perhaps a better way is to promote runtime configuration from the application code (not a config script) as a recommended practice. This can’t handle all scenarios, but I believe that it can handle most of them, and that it’s a better approach.
All that said, the proposed changes look good to me, as they seem fairly limited, but as a benefit they should make hacks such as replace_os_vars obsolete, and they should in general bring OTP releases and local development closer.
bitwalker
We want to address the second point but the question is: how do we control what is evaluated during boot/release? The initial proposal is about evaluating the exact same config files that we evaluate during a release. However, as we have seen examples in this thread, folks may even call System.cmd/2 in their config files. Simply executing all of the config files inside the release is prone to cause issues. A better approach would be if we could explicitly say what runs on boot. That’s what on_boot is about.
The reason we have a compile-time/run-time dichotomy in the first place is because people couldn’t use their Mix configs in a release - but that is where virtually everyone starts, and they find out the hard way that, oh actually, you can’t do that, because reasons. My take on the situation is that people already assume that configuration is unified by default, and only change that perspective once they have had to go to production. I think it represents not only an easier system to learn, but a pretty painless thing for existing applications to adapt to (anyone already deploying releases would be unaffected by these changes, barring things like invoking git inside their config.exs, and I think such things could be dealt with via warnings as part of the transition for this process (you’ve already invited discussion on that topic originally).
In other words, I think releases should be in a position to work like Mix with regards to configuration (evaluating them during boot). I don’t think there is value in two different sets of config files, or in a special DSL for wrapping “boot-only” config bits. Those things should be pushed into application code then, not defined in config.exs.
If many are talking about those points, it is a very strong sign that it is a problem in the scope of the proposal. When we introduced overridable and optional callbacks, there were a lot of confusion, so we broke the proposal in two and everything was clearer. So in my opinion something has to be done, even if that something is rewording the proposal or breaking it apart.
Not to diminish anyone’s complaints, but having been dealing with every shade of configuration problem with releases for years now, I have seen far more people stumble against the fact that config.exs did two different things between dev and releases, than that it doesn’t do two different things. I think the current confusion in this thread is a symptom of the solution we’ve had in place for a very long time now, and that this discussion is derailing into solving a problem which shouldn’t exist in the first place. To put it more plainly, I think that by unifying configuration (i.e. one config file, one set of semantics), it becomes easier to understand, and is easy to adapt to for older systems.
Configuration should be mostly static in nature - if people need to execute “boot-only” code, it should go in their applications, not in the config files. It is not clear to me what else would fall under this umbrella and not belong in your application code.
I’m not sure I agree with the premise that either of these are true and fall under the umbrella of “configuration”. Do we have some examples?
Popular in Proposals
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"