fceruti

fceruti

How to normalize function params

Hey everyone!

Saša Jurić on his talk called Clarity @ ElixirConf EU 2021, mentions a way of architecting your code that uses a function that he names normalize/2, but sadly, he says he doesn’t have time to share it.

I want to implement this programming style, but I’m missing this key ingredient. Do you know any libraries or gist that accomplish this? Thanks :pray:

def register(conn, params) do
  schema = [
    email: {:string, required: true},
    password: {:string, required: true}
    date_of_birth: :datetime,
    # ...
  ]

  with {:ok, params} <- normalize(params, schema),
    {:ok, user} <- MySystem.register(params) do
    # respond success
  else
  {:error, reason} ->
  # respond error
  end
end

update: if it’s not clear, I’m looking for a way of checking the existence and type of all the fields specified in schema.

Marked As Solved

riebeekn

riebeekn

I’ve not used it myself, but the tarams library looks like it might handle your use case if you want to go with a library instead of a custom solution GitHub - bluzky/tarams: Casting and validating external data and request parameters in Elixir and Phoenix

Also Liked

sasajuric

sasajuric

Author of Elixir In Action

Yes, on both accounts. I can’t share the code (it’s owned by the clients), but the basic take is something like

def normalize(data, types) do
  {%{}, types}
  |> Ecto.Changeset.cast(data, Map.keys(types))
  |> Ecto.Changeset.apply_action(:insert)
end

This is probably not enough (e.g. you may want to handle nils and missing keys), but it’s a solid start.

tomekowal

tomekowal

I wanted to reply on Twitter but I saw there is already a solution in here. I’d like to share my anyway :slight_smile:
Since we need to traverse the schema many times, I normalize it first and then use list comprehensions like this:

  def parse(params, schema) do
    # First I want to have entire schema in one format [{key, {type, opts}}]
    normalized_schema = for {key, type_spec} <- schema, do: {key, apply_default_opts(type_spec)}
    keys = for {key, _} <- normalized_schema, do: key
    types = for {key, {type, _opts}} <- normalized_schema, into: %{}, do: {key, type}
    required_fields = for {key, {_type, opts}} <- normalized_schema, Keyword.get(opts, :required), do: key
    defaults = for {key, {_type, opts}} <- normalized_schema, default = Keyword.get(opts, :default), into: %{}, do: {key, default}

    {defaults, types}
    |> cast(params, keys)
    |> validate_required(required_fields)
    |> apply_action(:normalize)
  end

  @default_opts [required: false]
  defp apply_default_opts(type) when is_atom(type), do: {type, @default_opts}
  defp apply_default_opts({type, opts}), do: {type, Keyword.merge(@default_opts, opts)}

The line computing defaults might be tricky to understand because it is long and introduces a variable inside the comprehension filter.
Also, I’d vote for naming that function “parse” instead of normalize. Parsing is an action of taking a bunch of data and trying to transform it into a structure that is understandable. In the spirit of Parse, don’t validate

It would be possible to push the schema specification even more to introduce validations like:
email: {:string, format: ~r/@/}
and then call Changeset.validate_format(changeset, key, format) for each. But that would require another option for each Ecto.Changeset validatior. It might be an overkill but the schema format is very readable so it is tempting :smiley:

stefanchrobot

stefanchrobot

Just a few remarks if you don’t mind:

Enum.reject(schema, fn {_, v}

I’d go with something longer and less generic than k (key?) and v (value?). Maybe {field, schema}? Plus in this case I think it makes sense to use {_k, v} to explain the meaning of the first element.

cond do
  is_tuple(v) -> {k, elem(v, 0)}
  is_atom(v) -> {k, v}
  true -> raise(ArgumentError, "Bad formed schema")
end

Using more pattern matching would be more idiomatic:

case v do
  {type, _opts} when is_atom(type) -> {k, type}
  type when is_atom(type) -> {k, type}
  _ -> raise ArgumentError, "bad schema: #{inspect(v)}"
end

And one more thing:

schema |> Enum.map(fn {k, v} -> ... end) |> Enum.into(%{})

can be replaced with:

Map.new(schema, fn {k, v} -> ... end)
fceruti

fceruti

Noted. Definitely it’s a little cryptic.

Oh yeah, the code looks much better.

I’m not convinced of this point. I’ve taken most of your ideas, but I kinda like normalize. To me parse means the data is going to change type.

Anyways, here is the revisited version with my take on @stefanchrobot and @tomekowal suggestions

defmodule Timetask.Helpers.Normalizer do
  import Ecto.Changeset

  @doc """
  Normalizes and validates that `params` is formed according to `schema`.

  ## Examples

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{name: "only required field"})
      {:ok, %{count: 10, name: "only required field"}}

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{"name" => "Also accepts strings as key"})
      {:ok, %{count: 10, name: "Also accepts strings as key"}}

      iex> normalized_params = Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: {:string, required: true},
      ...>     description: :string,
      ...>     count: {:integer, default: 10}
      ...>  }, %{description: "has no name"})
      ...> {:error, %{errors: errors}} = normalized_params
      ...> assert Keyword.has_key?(errors, :name)

      iex> Timetask.Helpers.Normalizer.normalize(%{
      ...>     name: "I'm a string"
      ...>  }, %{name: "Use atom or tuple"})
      ** (ArgumentError) Bad formed schema

  """
  @spec normalize(map, map) :: {:error, Ecto.Changeset.t()} | {:ok, map}
  def normalize(%{} = schema, %{} = params) do
    normalized_schema =
      for {field_name, type_spec} <- schema,
          do: {field_name, apply_default_opts(type_spec)}

    defaults =
      for {field_name, {_type, opts}} <- normalized_schema,
          default = Keyword.get(opts, :default),
          into: %{},
          do: {field_name, default}

    types =
      for {field_name, {type, _opts}} <- normalized_schema, into: %{}, do: {field_name, type}

    normalized_params =
      for {param_name, param_value} <- params, into: %{}, do: {to_atom(param_name), param_value}

    fields = for {field_name, _} <- normalized_schema, do: field_name

    required_fields =
      for {field_name, {_type, opts}} <- normalized_schema,
          Keyword.get(opts, :required),
          do: field_name

    {defaults, types}
    |> cast(normalized_params, fields)
    |> validate_required(required_fields)
    |> apply_action(:normalize)
  end

  defp to_atom(name) when is_atom(name), do: name
  defp to_atom(name) when is_bitstring(name), do: String.to_atom(name)
  defp to_atom(_), do: raise(ArgumentError, "Bad formed schema")

  @default_opts [required: false]
  defp apply_default_opts(type) when is_atom(type), do: {type, @default_opts}
  defp apply_default_opts({type, opts}), do: {type, Keyword.merge(@default_opts, opts)}
  defp apply_default_opts(_), do: raise(ArgumentError, "Bad formed schema")
end

vrcca

vrcca

I’d change String.to_atom by String.to_existing_atom, since this data comes from untrusted source.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New

We're in Beta

About us Mission Statement