ffloyd

ffloyd

Runtime-calculated default values for structs

The Problem

Currently, if I define a struct in the following way:

defmodule MyStruct do
  # Both x and y will have the FIXED values until next recompilation
  defstruct x: :rand.uniform(100), y: :rand.uniform(100)
end

I have defaults calculated at compile-time. This means that if I do %MyStruct{} or struct!(MyStruct), I will have the same default values for x and y until the next recompilation. This behavior is explicitly explained in Elixir’s documentation.

This compile-time evaluation has a “cascade effect” throughout the ecosystem. The most notable example is Ecto’s default option in schema field definitions. Another example is Oban structured jobs (their argument schema definitions). Such behavior likely exists because these libraries rely on defining struct defaults under the hood, or simply repeat the existing pattern.

Often, you don’t need defaults evaluated at runtime. After initially creating a schema or struct, a lot of code can be written before the first runtime default (e.g., start_of_period: DateTime.utc_now()) is needed. And that’s the point when the real problem starts:

  • In the case of a simple struct:
    • A developer needs to introduce a function like new()
    • … and change all the places where the struct is created via %Struct{...} to use this function
    • … and find all the places where the struct is created via struct!(mod_name) where mod_name is a variable that could theoretically be defined as an atom in config.exs, which makes such updates tricky.
  • In the case of an Ecto schema:
    • In addition to what’s needed for a simple struct, a developer needs to update every changeset used for creation
    • Alternatively, they can rely on autogenerate, but this only creates values just before insertion, so no business logic can rely on that default before the record is persisted
    • For virtual fields, autogenerate simply does not work, and the developer must track each place where data is read from the database in order to populate virtual fields with dynamic defaults.

In summary: it may require an inadequate amount of work for introducing a mere dynamic struct field default.

The Motivation

Recently, I analyzed a 2M+ line Elixir production codebase from the perspective of compilation determinism. This allowed me to automatically detect errors like default: DateTime.utc_now(). The reality is that even with documentation stating that such values are compile-time, people make this mistake frequently. This makes me think that “defaults are evaluated at compile-time” is not a feature anticipated by developers, but rather a trap that leads to weird bugs. Especially if the developer is new to Elixir.

Also, the fact that defining a pre-calculated and dynamic default requires completely different approaches seems weird to me.

The Proposal

I believe that one of the traits of a convenient programming language is this principle:

Things with similar semantics should have similar syntax.

For example, making things private in Elixir is done by adding p to a definer: defp, defmacrop. Another example is defining named “callables”: runtime callables are defined using def and defp, while compile-time callables are defined using defmacro and defmacrop. This consistent approach reduces cognitive load when working with the language.

So I propose:

defmodule MyStruct do
  defdynstruct x: :rand.uniform(100), y: :rand.uniform(100)
end

The same syntax rules apply, but each time such a struct is created (via struct/1, struct!/2, etc.), the defaults are recalculated. Note: making the %MyStruct{...} literal syntax work with dynamic defaults requires compiler changes—this is discussed in the Implementation section. If you want some values to be pre-calculated and others to be dynamic, you can still do it:

defmodule MyStruct do
  @default_x :rand.uniform(100)
  defdynstruct x: @default_x, y: :rand.uniform(100)
end

This will allow libraries like Ecto to change default behavior to support dynamic values. If a developer needs the previous compile-time behavior, they can use module attributes (as shown with @default_x above) to cache values at compile time.

The Implementation

I tried to fix it without patching Elixir. Below is what I implemented, and it works for most cases:

StructRuntimeDefaults module
defmodule StructRuntimeDefaults do
  @moduledoc """
  Provides runtime evaluation of default values for Elixir structs and Ecto schemas.

  The only thing it cannot handle is when `%StructName{...}` syntax is used for defining a value:
  Elixir compiler replaces it with the struct literal at compile time.

  Such calls should be avoided in favor of `struct!(StructName)` and `struct!(SomeName, ...)`.
  It's _okish_ to do it only in production code and leave tests and test factories to use `%StructName{}` syntax
  when diverse values are not important for testing.

  Such usage is easier to detect and refactor than finding all code paths that create the struct
  (e.g., via `struct/2`, `%StructName{}`, `Repo.get/3`, etc).

  !!! __Use this approach only when other approaches are insufficient!__ !!!

  ## Important Warning

  This implementation relies on Elixir and Ecto internals by overriding `__struct__/0`,
  `__struct__/1`, and `__schema__/1`. While these can be considered "mostly stable" interfaces, they are
  implementation details that could theoretically change in future versions.

  The Elixir compiler can occasionally miss auto-recompilation of modules that depend on the
  modified struct/schema module. After adding or changing runtime defaults, consider running
  a full recompilation if you notice that the changes are not taking effect.
  This problem has happened in the scope of test factories at least once.

  Static default values always take precedence over runtime defaults.

  ## Usage for Simple Structs

      defmodule MyStruct do
        use StructRuntimeDefaults

        defstruct other_data: [], current_time: nil

        struct_runtime_defaults(%{current_time: DateTime.utc_now()})
      end

  ## Usage for Ecto Schemas

      defmodule MySchema do
        use Ecto.Schema
        use StructRuntimeDefaults

        schema "my_table" do
          field :created_date, :date
          timestamps()
        end

        ecto_schema_runtime_defaults(%{created_date: Date.utc_today()})
      end
  """

  defmacro __using__(_ \\ []) do
    quote do
      import unquote(__MODULE__), only: [struct_runtime_defaults: 1, ecto_schema_runtime_defaults: 1]
    end
  end

  defmacro struct_runtime_defaults(defaults) do
    quote do
      # These are the Single Source of Truth (SSOT) for struct creation in Elixir
      # https://github.com/elixir-lang/elixir/blob/31905ca1364c8b841b49f68b7d2cbacbacb7de02/lib/elixir/lib/kernel.ex#L5626
      defoverridable __struct__: 0
      defoverridable __struct__: 1

      def __struct__ do
        Map.merge(super(), unquote(defaults))
      end

      def __struct__(kv) do
        original = super(kv)
        runtime_defaults = unquote(defaults)

        Map.merge(original, runtime_defaults, fn
          _key, nil, rt_def_val -> rt_def_val
          _key, orig_val, _rt_def_val -> orig_val
        end)
      end
    end
  end

  defmacro ecto_schema_runtime_defaults(defaults) do
    quote do
      struct_runtime_defaults(unquote(defaults))

      # Ecto schema support
      #
      # Ecto uses __schema__(:loaded) to initialize structs when loading from DB
      # it's critical to override this to apply runtime defaults on load when _virtual fields_ are involved.
      # https://github.com/elixir-ecto/ecto/blob/2bdbcb6a2c3022ae931ccb9c3e1920596a2da68a/lib/ecto/schema/loader.ex?plain=1#L12
      #
      # Unfortunately, overriding __schema__/1 was not enough at the time of writing because Ecto
      # relies on its own metadata about fields to set defaults and did not rely on __struct__/0 or __struct__/1 in some cases.
      defoverridable __schema__: 1

      def __schema__(atom) do
        case atom do
          :loaded ->
            original = super(:loaded)
            runtime_defaults = unquote(defaults)

            Map.merge(original, runtime_defaults, fn
              _key, nil, rt_def_val -> rt_def_val
              _key, orig_val, _rt_def_val -> orig_val
            end)

          _ ->
            super(atom)
        end
      end
    end
  end
end

Working on this allowed me to understand how dynamic defaults can be achieved. In the current Elixir implementation, the __struct__/0 and __struct__/1 functions are the Single Source of Truth for struct creation. However, AST with default values is evaluated at compile-time. If we change it to be evaluated dynamically inside __struct__(...), we will achieve the desired behavior. The module above proves that it works. With one exception though: the %Struct{...} syntax.

When the Elixir compiler sees %Struct{...} in the code, it runs __struct__(...) at compile-time and puts the calculated value directly into the final code. This behavior cannot be overridden without modifying the Elixir compiler itself. It means that it’s not possible to make universal “dynamic default values for structs” without changing the Elixir compiler.

I’m happy to try creating a PR for Elixir, but I need a green light from the community and core team. I also need alignment on the desired syntax.

So, implementing it for Elixir looks like merely defining defdynstruct that

  • mostly works like defstruct, but
  • places default values evaluation inside __struct__
  • and marks such structs to not be inlined as calculated literals by the compiler

I expect low performance cost, a small amount of code for implementation, and no impact on existing Elixir codebases.

Next Steps

Before proposing this to the Elixir core team mailing list, I want to collect community feedback. I’m specifically seeking input on:

  1. Syntax: do you like proposed defdynstruct syntax?
  2. Missing use cases: Does this solve the problem? Do you see situations where proposed approach is not enough?
  3. Missing implementation concerns: What potential issues or edge cases are missed in the proposal? I found %Struct{} syntax, are there others?
  4. Compatibility: Are there any compatibility concerns that I missed?

Let me know your thoughts! And if you simply like the idea, like the post to show support. Thank you!

Most Liked

LostKobrakai

LostKobrakai

Tbh I disagree with the idea of Repo.insert(%Reminder{}) needing to support dynamic defaults as well. Elixir is very much about being explicit. If you need your data to come from side effects I expect the code to tell me. %Reminder{} |> with_default_target_datetime() |> Repo.insert() is a lot better code than Repo.insert(%Reminder{}) automatically messing around with struct fields.

You’re argueing that struct defaults being a compile time thing prevent libraries from introducing dynamic field defaults. I’d counter that with dynamic defaults being something you want to have an explicit api for and once you do there’s no longer any limitation. I don’t expect ecto ever wanted to provide you with a dynamic defaults feature to begin with.

In the end your argument comes down to “I want to introduce dynamically calculated default values without changing my codebase” – not a great argument imo, such a change should be made explicit by changing code.

LostKobrakai

LostKobrakai

I don’t think we’ll find any alignment here. I still stand by the fact that structs should behave like native datatypes, while you seem to want to make them even more a special case to what they already are and partially out of necessity. I see %Struct{} as a notation to define data and data doesn’t execute code. Yes one can change the definition of the struct, but generally people try to avoid having the before and after of those definitions meet.

Execution is btw another footgun to your proposal. %Struct{} executing code means different runtime behaviour. It’s no longer “O(1)”, but could start an http request, fail somewhere on the path and raise some arbitrary exception crashing a whole process – just from %Struct{}.

christhekeele

christhekeele

Could throw together a custom credo check that warns on %Struct{...} when Struct.new/build is detected, perhaps. A community consensus on new vs build could be handy as well.

LostKobrakai

LostKobrakai

Personally I’m not a fan of this proposal. Structs are a tool for user defined types and those imo should align with the behaviour of native types, while allowing users to add their own names. None of the native datatypes like integer or list or map come with any sideeffects on creation, so why should structs do? All of those native datatypes need to have their creation code replaced once you need their values to be dynamically determined at creation time. Yes the implementation of structs would allow for more dynamic behaviour, but in an ideal world those implementation details wouldn’t exist in the first place. They’re just a crutch because the beam doesn’t have user defined types at the vm level - hence erlang doing essentially the same with tuple based records.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Yes. For me the split I’m observing comes down to people who feel that code which calls functions should be a function call, and this proposal where doing literal syntax is actually sometimes a way to call functions dynamically. You can tell from this framing which side I’m on I suppose lol.

But more seriously some of the objections to using new such as the time required to convert code base don’t hold water. Sure we are talking about dynamic data now but what about invariants? Should struct literal syntax also run validators on the values? The answer is: no, we already crossed this bridge with changesets. (And no the type system won’t solve this, there are invariants not amenable to types).

If we had a convention around new or build then this would be a non issue. You wouldn’t have 100 spots in your code to refactor you’d have 100 function calls and you make one change to the function. This is already what you have when using changesets.

Where Next?

Popular in Proposals: Ideas Top

jakeprem
Goal: To make JS.patch and JS.navigate more interoperable with JS.push. Scenario: Imagine making a reusable Phoenix component and you wa...
New
martosaur
TL;DR Logger.Translator acts as a global filter and swallows structure of some OTP reports, which some logger handlers could benefit from...
New
manhvu
In a large repo, working with module need to add alias too much is quite annoyed and not good for organizing code. I think better add su...
New
Asd
Process.resolve/1 What it does Takes any possible name of the process and returns a pid of the process or port (or nil if none found). Si...
New
rhcarvalho
Hi all, I would like to gather some feedback before a more intentional proposal to add a new :depth option when specifying a Git depende...
New
cevado
Being able to build nested relations in a schemaless changeset would be helpful to deal with complex forms on Phoenix without the need to...
New
ffloyd
The Problem Currently, if I define a struct in the following way: defmodule MyStruct do # Both x and y will have the FIXED values unti...
New
winsalva
Hello all. First of all i’m running this using termux on an android phone. Running mix assets.setup shows this message 06:54:08.450 [de...
New
bartblast
This could resolve to {[a: 1, b: 2]}. Was it ever considered to allow such syntax? Notice this: {:abc, a: 1, b: 2} and this: my_fun(:abc,...
New
sezaru
When writing my code, I always find __MODULE__ very useful to use as alias of that module “inner dependencies”, ex: alias __MODULE__.{Im...
New

Other popular topics Top

_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New

We're in Beta

About us Mission Statement