kreiling.io

kreiling.io

Manually Load an Elixir Config File at Runtime

I’d like to discuss how to manually load Elixir configuration files at runtime.

Macro Intent

I want to load an elixir config file (something that looks like, for example, config/config.exs) at runtime, and load that into the application configuration.

Micro Intent

Ultimately, I want to write a Config.Provider implementation which fetches the contents of a secret stored in AWS secrets manager. I’d like for those contents to be an Elixir script with import Config at the top, followed by configuration defined the same way you would in a project’s config/config.exs file. I then want to load that configuration into the current (running!) application’s configuration.

Questions & Discussion

  • Let’s assume that I can load the secret configuration file from anywhere into a binary with an arbitrary Config.Provider implementation (ignore SecretsManager for now). How would I go about compiling it and loading it into the application configuration?
  • What do you think of this approach to secret management, at least in theory?
  • What (at least roughly) do you do for secret management in your Elixir and/or Phoenix applications?

My Take

Let’s consider an alternative Config.Provider implementation: instead load a YAML file, parse it, and translate it into a keyword list to merge into the current config. In my eyes that has a few downsides that I dislike:

  1. Added dependency on a YAML parser
  2. Designing how that YAML is structured is not a trivial task
  3. It’s a roundabout way to get to what I ultimately want, which is just to load configuration from a remote location into the current node’s application configuration.

These are my thoughts. Let me know yours!

Most Liked

factoryd

factoryd

We run in ECS and use param store for our secrets. You can simply set environment variables in your task definition under secrets.

"containerDefinitions": [
      "secrets": [
        {
          "valueFrom": "/param/path",
          "name": "VARIABLE_NAME"
        }
]

from there I put them in prod.exs as configuration.

I know valueFrom supports secret manager too.

EDIT: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html

factoryd

factoryd

Oh I’m willing to brainstorm. I’m completely isolated and bored just like you! :stuck_out_tongue:

I do run as a release.

I actually just created a toy app to investigate possibilities right now.

kreiling.io

kreiling.io

So I’ve thought about it some more and looked at some docs in the Code module and there’s a warning about evaluating code coming from the network. Dang, seems like loading data from SecretsManager would be a bad idea from a security perspective…

Still - could there be another way??

I want to see if it’s possible. My current solution would be to have the config provider do:

defmodule MyConfigProvider do
  alias ExAws.SecretsManager, as: Secrets

  def init(secret_id) when is_binary(secret_id), do: secret_id

  def load(config, secret_id) do
    %{"SecretValue" => raw_elixir_code} = Secrets.get_secret_value(secret_id) |> ExAws.request!()
    {config_to_merge, _} = Code.eval_string(raw_elixir_code)
    Config.Reader.merge(config, config_to_merge)
  end
end
factoryd

factoryd

I have the same idea

defmodule SecretsConfigProvider do
  @behaviour Config.Provider

  def init(path) when is_binary(path), do: path

  def load(config, path) do
    {:ok, _} = Application.ensure_all_started(:jason)

    custom_config =
      path
      |> File.read!()
      |> Jason.decode!(keys: :atoms)
      |> resolve_config()

    Config.Reader.merge(config, custom_config)
  end

  defp resolve_config(config, acc \\ []) do
    Enum.into(config, [], fn
      {key, value} when is_map(value) -> {key, resolve_config(value)}
      {key, value} -> {key, read_secret(value)}
    end)
  end

  @aws [region: "us-west-1"]

  defp read_secret(secret) do
    %{"Parameter" => %{"Value" => value}} =
      secret
      |> ExAws.SSM.get_parameter()
      |> ExAws.request!(@aws)

    value
  end
end

and then load with a json file:

{
  "app": {
    "a": "/insurance/settlement/jobs_url",
    "b": {
      "c": "/notifications/api/url"
    }
  }
}

like

iex(1)> SecretsConfigProvider.load([existing: :config, app: [d: :d]], "/Users/martin/code/tmp/runtime_config/priv/secrets.json")
[
  existing: :config,
  app: [
    d: :d,
    a: "https://REDACTED.execute-api.us-west-1.amazonaws.com/staging",
    b: [c: "https://REDACTED.com"]
  ]
]

I believe this would work and is totally feasible.

…and I may just start using this instead of task definition secrets! :wink:

EDIT: Fixed the duplicate issue

Where Next?

Popular in Questions Top

itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement