Qqwy

Qqwy

TypeCheck Core Team

Specify: Comfortable, Explicit, Multi-Layered Configuration Specifications - v0.7

After two long topics (1 , 2) dedicated to how to best configure your elixir application,
talking with a couple of people at ElixirConf.eu about the current state of configuring applications,
and recognizing some patterns (both good and bad) in the Elixir libraries I and others have written in the last couple of years,

here is my take on the challenge of configuring applications and libraries in Elixir:


logo-text

hex.pm version Build Status

Comfortable, Explicit, multi-Layered configuration specifications.

hex - docs - github

Specify is an Elixir library to standardize the method of doing configuration for your applications and libraries. Rather than trying to force configuration to happen in one way, it enables both library writers and library consumers to specify on multiple different levels how they want to configure something.

The main goals that Specify has in mind are:

  1. Configuration is based on looking at a stack of (also per-specification and globally configurable!) configuration sources, falling back to defaults written in the configuration specification.
  2. Overriding the values in this stack of configuration using plain arguments passed to a function is also always possible, circumventing the ‘singleton problem’ that many configurable things suffer from.
  3. It is very clear what values are supported for a given configuration field, since a validation/parsing function is named when specifying it.
  4. Documentation is automatically generated from a configuration specification, listing names, descriptions, parser functions and default values.
  5. Normalization: After a call tto YourConfig.load you end up with an Elixir struct containing the configuration, with all fields having passed validation and parsing, so you know exactly what fields are there to use and what kind of values to expect. This greatly reduces the number of weird ambiguous errors when something is misconfigured.
  6. Fail-fast: The library raises errors on missing required configuration, things that cannot be parsed and attempting to override unexistent fields. The library raises compilation warnings on missing documentation. The library logs at ‘error’-log-level when a source cannot be reached during an attempted configuration load.

Example

defmodule Cosette.CastleOnACloud do
  require Specify
  Specify.defconfig sources: [Specify.Providers.MixEnv, Specify.Providers.Process] do
    @doc "there are no floors for me to sweep"
    field :floors_to_sweep, :integer, default: 0

    @doc "there are a hundred boys and girls"
    field :amount_boys_and_girls, :integer, default: 100

    @doc "The lady all in white holds me and sings a lullaby"
    field :lullaby, :string

    @doc "Crying is usually not allowed"
    field :crying_allowed, :boolean, default: false
  end
end

Trying to load this using Cosette.CatleOnACloud.load() will fail because a mandatory field is missing.

iex> Cosette.CastleOnACloud.load
** (Specify.MissingRequiredFieldsError) Missing required fields for `Elixir.Cosette.CastleOnACloud`: `:lullaby`.
    (specify) lib/specify.ex:179: Specify.prevent_missing_required_fields!/3
    (specify) lib/specify.ex:147: Specify.load/2

However, by configuring it using

Application.put_env(Cosette.CastleOnACloud, :lullaby, "I love you very much")
# or, in your config/conf.exs file:
config Cosette.CastleOnACloud, lullaby: "I love you very much"

or instead, using the Process Dictionary source:

Process.put(Cosette.CastleOnACloud, :lullaby, "I love you very much")

or specifying an inline lullaby value as part of the overrides: argument:

iex> Cosette.CastleOnACloud.load(overrides: [lullaby: "I love you very much", crying_allowed: true])

we end up with:

%Cosette.CastleOnACloud{
  crying_allowed: true,
  floors_to_sweep: 0,
  lullaby: "I love you very much",
  amount_boys_and_girls: 100
}

Roadmap

Specify is not yet done. Most importantly, it needs some tests and some more effort to make sure that its interface is understandable not only to me but also to the rest of the community :stuck_out_tongue_winking_eye: .
Also, at the very least I still want to create an environment variables provider (both for .env-files and System.get_env).
I am fairly certain that its documentation and examples can be improved to be more understandable. Besides this, I would love to hear answers to the following practical questions:

Does Specify work with how you are used to doing configuration, or does it clash, and in what way?

I have followed along and read through both of afore-mentioned topics so I think/hope that Specify is able to be useful to most common use-cases. Of course, it is only a tool, and definitely is somewhat opinionated.
To make it as generally useful as possible, I very much would like to hear your suggestions/ideas/questions.

What would be the best way to read from environment variables?

Basically, how to deal with the mismatch of Elixir’s snake_case typing and environment variables that usually, but not necessarily, are in CONSTANT_CASE. I am currently thinking of adding additional options to the fields to define an environment variable name (which defaults to the uppercase field name, but is overridable), with an additional option to the defconfig call that will be prefixed to these names, but I am not yet entirely certain this is the best way.

All with all, it was a ton of fun to write Specify (it might have been my most involved macro- and metaprogramming in Elixir to date). Very eager for your feedback!

hex - docs - github


note: starting at v. 0.4.0, Confy has been renamed to Specify.

Most Liked

Qqwy

Qqwy

TypeCheck Core Team

Starting today, Confy has been renamed to Specify to be more clear in the main focus of the library, which is to make fields and field-types explicit, regardless of whether we are talking about app-wide configuration, user-created settings or single-function-call options. :slightly_smiling_face:

Specify v.0.4.0 has been released, which replaces Confy v.0.3.0.

Also, what is better than a beautiful library? A beautiful library that has its own logo!
logo-text_25percent

OvermindDL1

OvermindDL1

I’d require the provider to handle that. For PostgreSQL for example you can LISTEN in it to get updates pushed for example.

The GenServer could then decide to switch over to the new configuration, stop and start a new version of itself with the new configuration, or potentially even recompile a module if, for optimization reasons, the compilation process itself is affected by the configuration.

The fact the OTP runtime doesn’t already have something that sends an update message to registered receivers when an application environment value changes is still a bit astounding to me… ^.^;

Qqwy

Qqwy

TypeCheck Core Team

Version 0.6 has been released!

Since the last update on the forum, the following improvements have been made:

  • 0.5 - Adds the nonnegative_integer , positive_integer , nonnegative_float , positive_float and timeout builtin parsers.
  • 0.6 - Adds the mfa and function builtin parsers.

I have one question to the community at large, which is about parsers like the function one. Currently there is no way to restrict a field expecting a function to a particular arity. What would be a good API for this?

I am thinking about two possibilities (but maybe there is a third as well):

  • field(field_name, {:function, arity}, default: some_val)
  • field(field_name, function(arity), default: some_val)

Currently, we already have {:list, :element_parser} (e.g. {:list, :integer}). It would be nice to keep a similar format, but I am not entirely convinced on the former format’s structure. :thinking:

As for the roadmap: Currently the only thing that I’d consider blocking before v 1.0 would be the possibility to nest specifications. (And besides this resolving above issue, I guess).

OvermindDL1

OvermindDL1

Took a bit to read it, looks like a fantastic start!

So a few starter questions:

  1. You can make arbitrary configuration loaders and handlers?

  2. What about one that pulls from a database on some kind of update message so it updates the configurations in real time, is there a way to tell something like a gen_server built on this to accept this new configuration (say like changing the pool size in a database pool)?

  3. How do you differentiate between the different stages of configurations? I.E. Compile-time, Load-Time, and Run-Time (dynamic)?

Qqwy

Qqwy

TypeCheck Core Team

Yes :slight_smile:. The Confy.Provider protocol can be implemented by anything you want.

Currently, not. I am planning to, in a way similar to what Vapor does, offer the possibility to spin up a background ‘configuration change watcher’ process that will trigger a message when the configuration changes at runtime. (Unfortunately for most providers this will require polling).

The GenServer could then decide to switch over to the new configuration, stop and start a new version of itself with the new configuration, or potentially even recompile a module if, for optimization reasons, the compilation process itself is affected by the configuration.

Confy treats them in the same way. Configuration is only ever loaded when YourConfigModule.load(options \\ []) is called. At that time, Confy will look through the sources that you have configured, and for every field, decide which source has the highest priority.

So if you have the following configuration:

defmodule MyRepo do
  import Confy
  defconfig sources: [Confy.Providers.MixEnv, Confy.Providers.Process] do
    field :endpoint, :string, default: "localhost"
    field :port, :integer, default: 8086
  end
end

then when calling MyRepo.load(options) it will:

  • Take the default values of :endpoint and :port
  • Possibly override them by a value set in the Application configuration.
  • Possibly override them by a value set in the current process’ process dictionary.
  • Possibly override them by a value set in the overrides: key of the options passed to MyRepo.load.

The idea is that you call YourConfigModule.load once (or infrequently) and keep the struct it returns around in e.g. your GenServer state.This means that parsing of individual config fields has to be done only once, and that you end up with an explicit representation of the configuration. Also, because you end up with ‘just’ a struct, the logic that uses this configuration can be completely stateless and might be easier to test.

Where Next?

Popular in Libraries Top

Crowdhailer
Raxx is an alternative to Plug and is inspired by projects such as Rack(Ruby) and Ring(Clojure). 1.0-rc.1 is now available. To use it re...
New
nikokozak
Hello all, I’ve been working on Svonix - a library for quickly integrating Svelte components into Phoenix views. It’s a much-needed succ...
New
archan937
It is a well-know topic within the Elixir community: “To mock or not to mock? :)” Every alchemist probably has his / her own opinion con...
New
tmbb
I’ve published the first version of my Makeup library. It’s a syntax highlighter for Elixir in the spirit of Pygments, Currently it highl...
New
MRdotB
I needed to reuse React components from my Chrome extension in my Phoenix/LiveView backend. I noticed that for Svelte/Vue, there are live...
New
tmbb
PhoenixWS - Websockets over Phoenix Channels Source code on Github here: https://github.com/tmbb/phoenix_ws Phoenix channels are a great...
New
engineeringdept
I’ve just released the first version of Snap, an Elasticsearch client. It borrows ideas about application structure and process managemen...
New
bryanjos
Hi, I just published version 0.23.0 of Elixirscript. Most of the changes are around JavaScript interop now that Elixirscript uses the ...
New
mtrudel
Bandit is an HTTP server for Plug and WebSock apps. Bandit is written entirely in Elixir and is built atop Thousand Island. It can serve...
New
bluzky
You may know https://ui.shadcn.com/, a UI component library for React. I really love it’s design style and components. I’ve built some co...
381 12391 119
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

Sub Categories:

We're in Beta

About us Mission Statement