tomekowal

tomekowal

Change of behaviour in Elixir 1.18 macros: (ArgumentError) tried to unquote invalid AST

Hi!

I have a piece of code that parses some json and creates validations from them. In one place, I need to create a regular expression from string.

In Elixir 1.17 this worked

iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> quote do unquote(~r/#{regex_string}/) end |> Macro.to_string
"~r/[0-9]{4}/"

In Elixir 1.18, I get an error

iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> quote do unquote(~r/#{regex_string}/) end |> Macro.to_string
** (ArgumentError) tried to unquote invalid AST: ~r/[0-9]{4}/
Did you forget to escape term using Macro.escape/1?
    (elixir 1.18.2) src/elixir_quote.erl:542: :elixir_quote.argument_error/1
    iex:3: (file)

I thought, it might be because of interpolation, so I tried interpolating outside of quote:

iex(1)> regex_string = "[0-9]{4}"
"[0-9]{4}"
iex(2)> regex = ~r/#{regex_string}/
~r/[0-9]{4}/
iex(3)> quote do unquote(regex) end |> Macro.to_string
** (ArgumentError) tried to unquote invalid AST: ~r/[0-9]{4}/
Did you forget to escape term using Macro.escape/1?
    (elixir 1.18.2) src/elixir_quote.erl:542: :elixir_quote.argument_error/1
    iex:7: (file)

The error suggests using Macro.escape, but it doesn’t make sense to me. Unquote should already escape and indeed, I would the generated code will have AST of the code isntead of the code.

iex(1)> quote do ~r/[0-9]{4}/ end
{:sigil_r, [delimiter: "/", context: Elixir, imports: [{2, Kernel}]],
 [{:<<>>, [], ["[0-9]{4}"]}, []]}
iex(2)> regex = ~r/#{regex_string}/
~r/[0-9]{4}/
iex(3)> escaped = Macro.escape(regex)
{:%{}, [],
 [
   __struct__: Regex,
   opts: [],
   re_pattern: {:{}, [],
    [
      :re_pattern,
      0,
      0,
      0,
      <<69, 82, 67, 80, 109, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 255, 255, 255,
        255, 255, 255, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 0, ...>>
    ]},
   re_version: {"8.44 2020-02-12", :little},
   source: "[0-9]{4}"
 ]}
iex(3)> quote do unquote(escaped) end |> Macro.to_string
"%{\n  __struct__: Regex,\n  opts: [],\n  re_pattern:\n    {:re_pattern, 0, 0, 0,\n     \"ERCPm\\0\\0\\0\\0\\0\\0\\0\\x01\\0\\0\\0\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0@\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\x83\\0)n\\0\\0\\0\\0\\0\\0\\xFF\\x03\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0m\\0\\x04\\0\\x04x\\0)\\0\"},\n  re_version: {\"8.44 2020-02-12\", :little},\n  source: \"[0-9]{4}\"\n}"

I am starting to wonder if Elixir 1.18 became more strict or is it a bug in Elixir?

Marked As Solved

Eiji

Eiji

In that case I guess you can compile the regular expression in compile time, then inspect it to get a string with sigil and finally convert a string to the AST.

Here is a complete script I have prepared:

Mix.install([:ecto])

defmodule MyLib.Schema do
  @doc """
  Generates the changeset function.

  The `name` is a changeset function name. Default to `:changeset`.
  The `properties` is an `Elixir` map with `atom` keys.
  The `func` is an optional funtion that could be used by the developer to add other validations. 
  """
  defmacro changeset(name \\ :changeset, properties, func \\ nil) do
    quote bind_quoted: [func: func, module: __MODULE__, name: name, properties: properties] do
      pipe =
        properties
        |> module.from_properties(__MODULE__)
        |> module.pipe_func(func)

      # pipe
      # |> Macro.to_string()
      # |> Code.format_string!()
      # =>
      # struct
      # |> cast(params, [:four_digit_code])
      # |> validate_format(:four_digit_code, ~r/[0-9]{4}/)

      struct = Macro.var(:struct, __MODULE__)
      params = Macro.var(:params, __MODULE__)

      def unquote(name)(unquote(struct), unquote(params)) do
        unquote(pipe)
      end
    end
  end

  @doc false
  def from_properties(properties, module) do
    fields = Map.keys(properties)
    struct = Macro.var(:struct, module)
    params = Macro.var(:params, module)

    cast =
      quote do
        cast(unquote(params), unquote(fields))
      end

    pipe = ast_pipe(struct, cast)
    Enum.reduce(properties, pipe, &from_field_properties/2)
  end

  @supported_validators ~w[pattern]a

  defp from_field_properties({field, properties}, acc) do
    properties
    |> Map.take(@supported_validators)
    |> Enum.reduce(acc, fn {key, value}, acc ->
      ast_pipe(acc, validator(field, key, value))
    end)
  end

  defp validator(field, :pattern, value) do
    quote do
      validate_format(unquote(field), unquote(quoted_regex_sigil(value)))
    end
  end

  defp quoted_regex_sigil(source) do
    source
    # creates regular expression from source
    |> Regex.compile!()
    # inspect returns a string with a sigil
    |> inspect()
    # escaped AST form
    |> Code.string_to_quoted!()
  end

  def pipe_func(left, nil), do: left

  def pipe_func(left, func) do
    ast_pipe(
      left,
      quote do
        then(unquote(func))
      end
    )
  end

  defp ast_pipe(left, right) do
    quote do
      unquote(left) |> unquote(right)
    end
  end
end

defmodule MyApp.Schema do
  use Ecto.Schema

  import Ecto.Changeset
  import MyLib.Schema

  embedded_schema do
    field(:four_digit_code, :string)
  end

  changeset(%{four_digit_code: %{pattern: "[0-9]{4}"}})
end

defmodule Example do
  def sample do
    MyApp.Schema.changeset(%MyApp.Schema{}, %{four_digit_code: "0007"})
    # => %Ecto.Changeset{valid?: true}
  end
end

You should be able to easily adapt the example code to your needs.

Also Liked

tomekowal

tomekowal

Thank you!

This part is exactly what I was missing!

  defp quoted_regex_sigil(source) do
    source
    # creates regular expression from source
    |> Regex.compile!()
    # inspect returns a string with a sigil
    |> inspect()
    # escaped AST form
    |> Code.string_to_quoted!()
  end

I didn’t realise, Code has string_to_quoted!. That will ensure correct representation in the final generated code.

Brilliant answer!

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

We're in Beta

About us Mission Statement