New Version Released!

Version 0.2.4:

Features

This release adds runtime schemas. Runtime schemas are schemas that are defined at runtime and allow for casting to existing structs or to a bare map instead of a struct. This makes it really easy to integrate with Ecto for example to save an XML response into a db.

See the livebook for more details: data_schema/livebooks/runtime_schemas.livemd at main · Adzz/data_schema · GitHub

Here is a small example of what is possible:

defmodule User do
  use Ecto.Schema

  schema "users" do
    field :name, :string
    field :age, :integer
  end

  def update_details_from_xml(user_id, xml) do
    schema = [
      field: {:name, "/Response/User/@name", &{:ok, &1}},
      field: {:age, "/Response/User/@age", &parse_in/1t}
    ]

    with {:ok, changes} <- DataSchema.to_struct(xml, %{}, schema, XpathAccessor),
      %User{} = user <- Repo.get(user_id, User),
      %{valid?: true} = changeset <- Ecto.Changeset.change(%User{}, changes) do
      Repo.update(changeset)
    end
  end

  defp parse_int(string) do 
    case Integer.parse(string)  do
      {int, _} -> {:ok, int}
      _error -> :error
    end
  end

end

xml = """
<Response>
  <User name="Jeff" age="12" />
</Response>
"""
User.update_details_from_xml("123", xml)

I’m also now realising I don’t think I ever actually linked to the repo:

Great questions!

Validations

Can I run validations on my data?

Right now the focus is on parsing over validation. What I mean by that is instead of doing something like this:

input = %{"name" => ""}

input
|> DataSchema.to_struct(User)
|> validate_name_not_blank()

Or even:

input = %{"name" => ""}

input
|> validate_name_not_blank()
|> DataSchema.to_struct(User)

we can define our casting function to return an :error if it receives an empty string:

defmodule NonBlankString do
  @behaviour DataSchema.CastBehaviour

  @impl true
  def cast(""), do: {:error, "Field was blank!"}
  def cast(value), do: {:ok, to_string(value)}
end

defmodule User do
  import DataSchema, only: [data_schema: 1]

  data_schema([
    field: {:user, "user", NonBlankString}
  ])
end

My current take on validations is that they are for when you can’t design away the need for them (via making illegal states unrepresentable). So the idea is that the schema defines what is valid.

HOWEVER - as you can see in the above examples you could define your own functions before / after struct creation if you felt the need.

It’s possible that some validations can’t be expressed per field, in which case we could add some in the future.

Phoenix Forms

There is nothing specially added yet for phoenix forms, but off the top of my head there are a few ways you could approach it. One way is to use a schemaless changeset in the form:

  def index(conn, _params) do
    types = %{name: :string}
    user = %User{}
    changeset = Ecto.Changeset.change({user, types}, %{})
    render(conn, "index.html", changeset: changeset)
  end

# With a form like this
<%= form_for @changeset, Routes.user_path(@conn, :create), fn f -> %>
  <label>
    Name: <%= text_input f, :name %>
  </label>
  <%= submit "Submit" %>
<% end %>

Then when you post the form:

def create(conn, %{"user" => user_input}) do
  case DataSchema.to_struct(user_input, User) do
     {:error, error} -> ...
     {:ok, struct} -> ...
  end
end

We could possibly make this easier by supplying a function something like DataSchema.schemaless_changeset_from_schema(User):

  def index(conn, _params) do
    changeset = DataSchema.schemaless_changeset_from_schema(User)
    render(conn, "index.html", changeset: changeset)
  end

You’d also have to do the work of converting the error to a changeset error, which we could probably write some functions to help with, but it might be as easy as:

def create(conn, %{"user" => user_input}) do
  case DataSchema.to_struct(user_input, User) do
     {:error, %{errors: [{field,  message}]}} ->
       changeset =
         {%User{}, %{name: :string}}
         |> Ecto.Changeset.change(user_input)
         |> Ecto.Changeset.add_error(field, message)
         
         render(conn, changeset: changeset)
     {:ok, struct} ->
       render(...)
  end
end

My feel is that ecto might feel more natural, but open to the use case.

Looks nice, though to add to the discussion here are some alternative libraries also in this space:

• GitHub - vic/params: Easy parameters validation/casting with Ecto.Schema, akin to Rails' strong parameters.
• GitHub - bluzky/tarams: Casting and validating external data and request parameters in Elixir and Phoenix
• GitHub - akoutmos/pharams: Parameter validation for Elixir Phoenix Framework
• GitHub - satom99/paramus: Parameter validation for Phoenix

Just putting this out there, but… What I REALLY want is something similar to Ecto for JSON/XML…

Meaning I find myself needing to consume some JSON structure. eg:

{
    ....,
    "firewall": {
        "dnat": {
            "nat-in": [
                {
                    "dest": "loc:192.168.111.4",
                    "dport": "7",
                    "proto": "tcp",
                    "source": "net"
                },
                {
                    "dest": "loc:192.168.111.4:80",
                    "dport": "8080",
                    "origdest": "&ppp0",
                    "proto": "tcp",
                    "source": "net"
                }
            ]
        }
    },
    ....,
}

So this is a map of maps of maps, which contains an array of maps.

Now this snippet is part of a much larger JSON structure which has config for other stuff, ie there are other keys at the top level with their own trees under.

Now I want to parse chunks of this into Elixir structures, check that it’s valid before starting, present those to the user as some kind of phoenix/Liveview form, accept back the updated params and validate them (so that I can do instant errors on screen). Finally I want to be able to diff what changed from the original and re-apply it to the current JSON structure

Whilst I’ve been a little over specific on some of my own use case, I don’t think this is so different from a use case you likely have in mind: consume some API end point, present the details to the user, allow them to edit stuff, send the changes back to the API end point?

Things which might not be obvious from the above:

• I need to validate fields in combination with each other, so certain params may only be valid if the :action is something specific
• It’s REALLY boring mapping a map of maps of lists of maps into Ecto format… Ecto can only cope with representing database tables, so your map of maps ends up needing to become a list structure where you copy the keys in and out (think how you would represent it in an SQL database). It would be SOOO much easier if Ecto could understand something like a map structure in it’s “has_many” fields (yes you can do custom data types like {:map, string}, but then you lose the ability to use schemas and changesets on those fields)
• I need to also validate the keys of the maps. They need to be sanitised and controlled for length, etc (as they may map to UI elements, etc)
• I need some level of round trip ability
• I need to “diff” the changes (so I can apply only the changes back upstream - I want to be granular if simultaneous changes were made to separate parts of the document. However, ideally I want to be able to re-run my “is it valid” after re-apply those changes as two edits might individually not clash, but there might be dependencies between the key values, eg we might have a section for the IP range of the local network and another section for the DHCP parameters, but we have a schema validation between the two as we enforce that one is within the same range as the other.

I solve this at the moment using ecto changesets. The shape of a changeset can cross chunks of the whole json document if needed to enforce cross schema changes. ie a changeset “plucks out” a bunch of fields from the JSON input, and kind of flattens them into the structures allowed within ecto (lists). Then we can run our nested validations, etc. Then unfortunately this needs another function to reverse this process as it’s not necessarily purely mechanical to reverse the original extraction. It’s also painful to represent maps of maps as these need flattening into lists with an id column to represent the map key names (and this reversing later)

What I desire is something like a JSON parser, coupled to a generic structure validator. Which in turn can be used in phoenix forms with functioning error handling (the phoenix error function is something you define, so it can work with any library which produces a validation output including some per field error term)

Does this sound like a direction you are heading in?