ffloyd

ffloyd

Runtime-calculated default values for structs

The Problem

Currently, if I define a struct in the following way:

defmodule MyStruct do
  # Both x and y will have the FIXED values until next recompilation
  defstruct x: :rand.uniform(100), y: :rand.uniform(100)
end

I have defaults calculated at compile-time. This means that if I do %MyStruct{} or struct!(MyStruct), I will have the same default values for x and y until the next recompilation. This behavior is explicitly explained in Elixir’s documentation.

This compile-time evaluation has a “cascade effect” throughout the ecosystem. The most notable example is Ecto’s default option in schema field definitions. Another example is Oban structured jobs (their argument schema definitions). Such behavior likely exists because these libraries rely on defining struct defaults under the hood, or simply repeat the existing pattern.

Often, you don’t need defaults evaluated at runtime. After initially creating a schema or struct, a lot of code can be written before the first runtime default (e.g., start_of_period: DateTime.utc_now()) is needed. And that’s the point when the real problem starts:

  • In the case of a simple struct:
    • A developer needs to introduce a function like new()
    • … and change all the places where the struct is created via %Struct{...} to use this function
    • … and find all the places where the struct is created via struct!(mod_name) where mod_name is a variable that could theoretically be defined as an atom in config.exs, which makes such updates tricky.
  • In the case of an Ecto schema:
    • In addition to what’s needed for a simple struct, a developer needs to update every changeset used for creation
    • Alternatively, they can rely on autogenerate, but this only creates values just before insertion, so no business logic can rely on that default before the record is persisted
    • For virtual fields, autogenerate simply does not work, and the developer must track each place where data is read from the database in order to populate virtual fields with dynamic defaults.

In summary: it may require an inadequate amount of work for introducing a mere dynamic struct field default.

The Motivation

Recently, I analyzed a 2M+ line Elixir production codebase from the perspective of compilation determinism. This allowed me to automatically detect errors like default: DateTime.utc_now(). The reality is that even with documentation stating that such values are compile-time, people make this mistake frequently. This makes me think that “defaults are evaluated at compile-time” is not a feature anticipated by developers, but rather a trap that leads to weird bugs. Especially if the developer is new to Elixir.

Also, the fact that defining a pre-calculated and dynamic default requires completely different approaches seems weird to me.

The Proposal

I believe that one of the traits of a convenient programming language is this principle:

Things with similar semantics should have similar syntax.

For example, making things private in Elixir is done by adding p to a definer: defp, defmacrop. Another example is defining named “callables”: runtime callables are defined using def and defp, while compile-time callables are defined using defmacro and defmacrop. This consistent approach reduces cognitive load when working with the language.

So I propose:

defmodule MyStruct do
  defdynstruct x: :rand.uniform(100), y: :rand.uniform(100)
end

The same syntax rules apply, but each time such a struct is created (via struct/1, struct!/2, etc.), the defaults are recalculated. Note: making the %MyStruct{...} literal syntax work with dynamic defaults requires compiler changes—this is discussed in the Implementation section. If you want some values to be pre-calculated and others to be dynamic, you can still do it:

defmodule MyStruct do
  @default_x :rand.uniform(100)
  defdynstruct x: @default_x, y: :rand.uniform(100)
end

This will allow libraries like Ecto to change default behavior to support dynamic values. If a developer needs the previous compile-time behavior, they can use module attributes (as shown with @default_x above) to cache values at compile time.

The Implementation

I tried to fix it without patching Elixir. Below is what I implemented, and it works for most cases:

StructRuntimeDefaults module
defmodule StructRuntimeDefaults do
  @moduledoc """
  Provides runtime evaluation of default values for Elixir structs and Ecto schemas.

  The only thing it cannot handle is when `%StructName{...}` syntax is used for defining a value:
  Elixir compiler replaces it with the struct literal at compile time.

  Such calls should be avoided in favor of `struct!(StructName)` and `struct!(SomeName, ...)`.
  It's _okish_ to do it only in production code and leave tests and test factories to use `%StructName{}` syntax
  when diverse values are not important for testing.

  Such usage is easier to detect and refactor than finding all code paths that create the struct
  (e.g., via `struct/2`, `%StructName{}`, `Repo.get/3`, etc).

  !!! __Use this approach only when other approaches are insufficient!__ !!!

  ## Important Warning

  This implementation relies on Elixir and Ecto internals by overriding `__struct__/0`,
  `__struct__/1`, and `__schema__/1`. While these can be considered "mostly stable" interfaces, they are
  implementation details that could theoretically change in future versions.

  The Elixir compiler can occasionally miss auto-recompilation of modules that depend on the
  modified struct/schema module. After adding or changing runtime defaults, consider running
  a full recompilation if you notice that the changes are not taking effect.
  This problem has happened in the scope of test factories at least once.

  Static default values always take precedence over runtime defaults.

  ## Usage for Simple Structs

      defmodule MyStruct do
        use StructRuntimeDefaults

        defstruct other_data: [], current_time: nil

        struct_runtime_defaults(%{current_time: DateTime.utc_now()})
      end

  ## Usage for Ecto Schemas

      defmodule MySchema do
        use Ecto.Schema
        use StructRuntimeDefaults

        schema "my_table" do
          field :created_date, :date
          timestamps()
        end

        ecto_schema_runtime_defaults(%{created_date: Date.utc_today()})
      end
  """

  defmacro __using__(_ \\ []) do
    quote do
      import unquote(__MODULE__), only: [struct_runtime_defaults: 1, ecto_schema_runtime_defaults: 1]
    end
  end

  defmacro struct_runtime_defaults(defaults) do
    quote do
      # These are the Single Source of Truth (SSOT) for struct creation in Elixir
      # https://github.com/elixir-lang/elixir/blob/31905ca1364c8b841b49f68b7d2cbacbacb7de02/lib/elixir/lib/kernel.ex#L5626
      defoverridable __struct__: 0
      defoverridable __struct__: 1

      def __struct__ do
        Map.merge(super(), unquote(defaults))
      end

      def __struct__(kv) do
        original = super(kv)
        runtime_defaults = unquote(defaults)

        Map.merge(original, runtime_defaults, fn
          _key, nil, rt_def_val -> rt_def_val
          _key, orig_val, _rt_def_val -> orig_val
        end)
      end
    end
  end

  defmacro ecto_schema_runtime_defaults(defaults) do
    quote do
      struct_runtime_defaults(unquote(defaults))

      # Ecto schema support
      #
      # Ecto uses __schema__(:loaded) to initialize structs when loading from DB
      # it's critical to override this to apply runtime defaults on load when _virtual fields_ are involved.
      # https://github.com/elixir-ecto/ecto/blob/2bdbcb6a2c3022ae931ccb9c3e1920596a2da68a/lib/ecto/schema/loader.ex?plain=1#L12
      #
      # Unfortunately, overriding __schema__/1 was not enough at the time of writing because Ecto
      # relies on its own metadata about fields to set defaults and did not rely on __struct__/0 or __struct__/1 in some cases.
      defoverridable __schema__: 1

      def __schema__(atom) do
        case atom do
          :loaded ->
            original = super(:loaded)
            runtime_defaults = unquote(defaults)

            Map.merge(original, runtime_defaults, fn
              _key, nil, rt_def_val -> rt_def_val
              _key, orig_val, _rt_def_val -> orig_val
            end)

          _ ->
            super(atom)
        end
      end
    end
  end
end

Working on this allowed me to understand how dynamic defaults can be achieved. In the current Elixir implementation, the __struct__/0 and __struct__/1 functions are the Single Source of Truth for struct creation. However, AST with default values is evaluated at compile-time. If we change it to be evaluated dynamically inside __struct__(...), we will achieve the desired behavior. The module above proves that it works. With one exception though: the %Struct{...} syntax.

When the Elixir compiler sees %Struct{...} in the code, it runs __struct__(...) at compile-time and puts the calculated value directly into the final code. This behavior cannot be overridden without modifying the Elixir compiler itself. It means that it’s not possible to make universal “dynamic default values for structs” without changing the Elixir compiler.

I’m happy to try creating a PR for Elixir, but I need a green light from the community and core team. I also need alignment on the desired syntax.

So, implementing it for Elixir looks like merely defining defdynstruct that

  • mostly works like defstruct, but
  • places default values evaluation inside __struct__
  • and marks such structs to not be inlined as calculated literals by the compiler

I expect low performance cost, a small amount of code for implementation, and no impact on existing Elixir codebases.

Next Steps

Before proposing this to the Elixir core team mailing list, I want to collect community feedback. I’m specifically seeking input on:

  1. Syntax: do you like proposed defdynstruct syntax?
  2. Missing use cases: Does this solve the problem? Do you see situations where proposed approach is not enough?
  3. Missing implementation concerns: What potential issues or edge cases are missed in the proposal? I found %Struct{} syntax, are there others?
  4. Compatibility: Are there any compatibility concerns that I missed?

Let me know your thoughts! And if you simply like the idea, like the post to show support. Thank you!

Most Liked

LostKobrakai

LostKobrakai

Tbh I disagree with the idea of Repo.insert(%Reminder{}) needing to support dynamic defaults as well. Elixir is very much about being explicit. If you need your data to come from side effects I expect the code to tell me. %Reminder{} |> with_default_target_datetime() |> Repo.insert() is a lot better code than Repo.insert(%Reminder{}) automatically messing around with struct fields.

You’re argueing that struct defaults being a compile time thing prevent libraries from introducing dynamic field defaults. I’d counter that with dynamic defaults being something you want to have an explicit api for and once you do there’s no longer any limitation. I don’t expect ecto ever wanted to provide you with a dynamic defaults feature to begin with.

In the end your argument comes down to “I want to introduce dynamically calculated default values without changing my codebase” – not a great argument imo, such a change should be made explicit by changing code.

LostKobrakai

LostKobrakai

I don’t think we’ll find any alignment here. I still stand by the fact that structs should behave like native datatypes, while you seem to want to make them even more a special case to what they already are and partially out of necessity. I see %Struct{} as a notation to define data and data doesn’t execute code. Yes one can change the definition of the struct, but generally people try to avoid having the before and after of those definitions meet.

Execution is btw another footgun to your proposal. %Struct{} executing code means different runtime behaviour. It’s no longer “O(1)”, but could start an http request, fail somewhere on the path and raise some arbitrary exception crashing a whole process – just from %Struct{}.

christhekeele

christhekeele

Could throw together a custom credo check that warns on %Struct{...} when Struct.new/build is detected, perhaps. A community consensus on new vs build could be handy as well.

LostKobrakai

LostKobrakai

Personally I’m not a fan of this proposal. Structs are a tool for user defined types and those imo should align with the behaviour of native types, while allowing users to add their own names. None of the native datatypes like integer or list or map come with any sideeffects on creation, so why should structs do? All of those native datatypes need to have their creation code replaced once you need their values to be dynamically determined at creation time. Yes the implementation of structs would allow for more dynamic behaviour, but in an ideal world those implementation details wouldn’t exist in the first place. They’re just a crutch because the beam doesn’t have user defined types at the vm level - hence erlang doing essentially the same with tuple based records.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Yes. For me the split I’m observing comes down to people who feel that code which calls functions should be a function call, and this proposal where doing literal syntax is actually sometimes a way to call functions dynamically. You can tell from this framing which side I’m on I suppose lol.

But more seriously some of the objections to using new such as the time required to convert code base don’t hold water. Sure we are talking about dynamic data now but what about invariants? Should struct literal syntax also run validators on the values? The answer is: no, we already crossed this bridge with changesets. (And no the type system won’t solve this, there are invariants not amenable to types).

If we had a convention around new or build then this would be a non issue. You wouldn’t have 100 spots in your code to refactor you’d have 100 function calls and you make one change to the function. This is already what you have when using changesets.

Where Next?

Popular in Proposals: Ideas Top

snofang
In a typical business development task, having a function in a context module which accepts attributes of map type and passes them to Ect...
New
MeerKatDev
many times we do stuff like (e.g. test setups, json views in phoenix) aaa = ... bbb = ... ccc = ... %{aaa: aaa, bbb: bbb, ccc: ccc} and...
New
Asd
Process.resolve/1 What it does Takes any possible name of the process and returns a pid of the process or port (or nil if none found). Si...
New
jakeprem
Goal: To make JS.patch and JS.navigate more interoperable with JS.push. Scenario: Imagine making a reusable Phoenix component and you wa...
New
kip
Sumary of proposal DateTime.from_iso8601/3 adjusted to: set all numeric fields to the values as parsed (not shifted to UTC), preservi...
New
mudasobwa
I am not sure it deserves to be discussed in the mailing list, so I’d start here. assert/1 and/or refute/1 macros print the following er...
New
dli
Ecto currently supports some data-modifying WITH statements / CTEs for Postgres: Options: […] :operation - one of :all , :update_all ,...
New
cheerfulstoic
I feel like Elixir is getting big enough and old enough that I’m starting to experience problems with conflicting dependencies. An examp...
New
Oliver
One common problem we face in constructing lists is that there is (AFAIK) no support for conditionally inserting members into list declar...
New
sezaru
When writing my code, I always find __MODULE__ very useful to use as alias of that module “inner dependencies”, ex: alias __MODULE__.{Im...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New

We're in Beta

About us Mission Statement