Very cool exploration @zachallaun!

Just some ideas (feel free to fully ignore them):

1. You can probably skip protocols and their implementations from lib_elixir. All of our protocols and their implementations are public, so it is very unlikely they will change between versions in an incompatible way.

2. Perhaps instead of allowing some modules to be removed, you could ask developers to list which modules they want to use, then you traverse their abstract code and find what they depend on, and convert these too, recursively. This means that Spitfire, which only really needs the tokenizer, gets the minimum stuff they need. You may have some corner cases, for example if we use some module conditionally, but then you can manually add those (and they should be few), such as the string_tokenizer used by the tokenizer.

Yep, that seems to work just fine.

As a quick update, here’s what it currently takes to get Spitfire using lib_elixir with all tests passing while running on Elixir 1.15 (cc @mhanberg):

diff --git a/mix.exs b/mix.exs
index fc7eabf..90f27ab 100644
--- a/mix.exs
+++ b/mix.exs
@@ -12,7 +12,9 @@ defmodule Spitfire.MixProject do
       start_permanent: Mix.env() == :prod,
       deps: deps(),
       docs: [main: "Spitfire"],
-      package: package()
+      package: package(),
+      compilers: [:lib_elixir] ++ Mix.compilers(),
+      lib_elixir: {Spitfire.LibElixir, "v1.17.2", [Code, Macro, Macro.Env, :elixir_tokenizer]}
     ]
   end
@@ -26,6 +28,7 @@ defmodule Spitfire.MixProject do
   # Run "mix help deps" to learn about dependencies.
   defp deps do
     [
+      {:lib_elixir, path: "../lib_elixir", runtime: false},
       {:ex_doc, ">= 0.0.0", only: :dev},
       {:styler, "~> 0.11", only: :dev}
       # {:dep_from_hexpm, "~> 0.3.0"},

diff --git a/lib/spitfire.ex b/lib/spitfire.ex
index 8e92c35..9174bc5 100644
--- a/lib/spitfire.ex
+++ b/lib/spitfire.ex
@@ -1989,7 +1989,7 @@ defmodule Spitfire do
     tokens =
       case code
            |> String.to_charlist()
-           |> :spitfire_tokenizer.tokenize(opts[:line] || 1, opts[:column] || 1, opts) do
+           |> :spitfire_lib_elixir_tokenizer.tokenize(opts[:line] || 1, opts[:column] || 1, opts) do
         {:ok, _, _, _, tokens} ->
           tokens

diff --git a/lib/spitfire/env.ex b/lib/spitfire/env.ex
index 50ee68d..bd76dcc 100644
--- a/lib/spitfire/env.ex
+++ b/lib/spitfire/env.ex
@@ -2,8 +2,14 @@ defmodule Spitfire.Env do
   @moduledoc """
   Environment querying
   """
+
+  alias Spitfire.LibElixir.Code
+  alias Spitfire.LibElixir.Macro
+
   @env %{
-    Macro.Env.prune_compile_info(__ENV__)
+    (Macro.Env
+     |> struct(Map.from_struct(__ENV__))
+     |> Macro.Env.prune_compile_info())
     | line: 0,
       file: "nofile",
       module: nil,

diff --git a/test/spitfire_test.exs b/test/spitfire_test.exs
index 7f0c6b2..30475ff 100644
--- a/test/spitfire_test.exs
+++ b/test/spitfire_test.exs
@@ -1,6 +1,8 @@
 defmodule SpitfireTest do
   use ExUnit.Case

+  alias Spitfire.LibElixir.Code
+
   doctest Spitfire

   describe "valid code" do

I’m not sure what you mean by “fixed the mistake” WRT Igniter, but I think I was just wrong when explaining our main use case with Spitfire. Sorry about that Error tolerance is why we use Spitfire.container_cursor_to_quoted/1 as opposed to Code.Fragment, but what we actually use Spitfire for is primarily Spitfire.Env.expand.

The way we are using it currently (and we aim to expand this usage) is, for example, when patching in a module name, we want it to respect existing module aliases.

For example, when installing AshPostgres, we make sure that your Repo module exists, is configured correctly, etc., and then we make sure that it is a child of your application (and that you have an application file, we create it if it doesn’t exist).

This patching logic is naive at the moment, but it will get more robust over time. I’ve also removed a bunch of stuff token this example simple

# in `Igniter.Project.Application`
  def do_add_child(igniter, application, to_supervise) do
    path = Igniter.Code.Module.proper_location(application)

    Igniter.update_elixir_file(igniter, path, fn zipper ->
      with {:ok, zipper} <- Igniter.Code.Module.move_to_module_using(zipper, Application),
           {:ok, zipper} <- Igniter.Code.Function.move_to_def(zipper, :start, 2),
           {:ok, zipper} <-
             Igniter.Code.Function.move_to_function_call_in_current_scope(
               zipper,
               :=,
               [2],
               fn call ->
                 Igniter.Code.Function.argument_matches_pattern?(
                   call,
                   0,
                   {:children, _, context} when is_atom(context)
                 ) &&
                   Igniter.Code.Function.argument_matches_pattern?(call, 1, v when is_list(v))
               end
             ) do
        zipper
        |> Zipper.down()
        |> Zipper.rightmost()
        |> Igniter.Code.List.append_new_to_list(Macro.escape(to_supervise), diff_checker)
      else
        _ ->
          {:warning, "...."}
      end
    end)
  end

So when using append_new_to_list, that ultimately ends up using code that expands the environment at the place you are bringing in code, and uses that to

1. honor aliases when checking for matches in the list
2. use aliases when inserting a module into the AST

Error tolerance can be useful for us because it is theoretically possible to introduce invalid AST “temporarily” while working with igniter. You compose a bunch of AST modifiers, and I didn’t want to necessarily guarantee that, after every single modification you make, the AST is valid. Only when actually writing the file must it be valid.

If its not valid, though, you won’t get the nice features of being able to determine env at a location.

This is the entirety of our Spitfire usage currently.

  @doc """
  Expands the environment at the current zipper position and returns the
  expanded environment. Currently used for properly working with aliases.
  """
  def current_env(zipper) do
    zipper
    |> do_add_code({:__cursor__, [], []}, :after, false)
    |> Zipper.topmost_root()
    |> Sourceror.to_string()
    |> String.split("__cursor__()", parts: 2)
    |> List.first()
    |> Spitfire.container_cursor_to_quoted()
    |> then(fn {:ok, ast} ->
      ast
    end)
    |> Spitfire.Env.expand("file.ex")
    |> then(fn {_ast, _final_state, _final_env, cursor_env} ->
      {:ok, struct(Macro.Env, cursor_env)}
    end)
  rescue
    e ->
      {:error, e}
  end

So, one thing to note here is that Igniter currently depends on spitfire, using it to provide context aware source-code patches. This also means that packages that expose Igniter installers or composable functions also depend on spitfire (i.e Ash). There are various reasons we couldn’t make this a dev-only dependency, primarily around DX/UX (DX being folks writing igniter-installers/tasks and UX being end users running them).

Ultimately this means that in order for anyone to compile their Ash application they’d have to be able to download an archive, and deal w/ the associated costs of having the new dependency, which naturally is going to be problematic for our use case. We’d have to stop using spitfire if it went that route.

It may be the case that Igniter needs its own thing like Spitfire. Or perhaps lib_elixir's work can be disabled in some way forcing dependencies to rely on the current elixir version instead. But then we’d be back to defensively writing code.

I don’t really have an answer here unfortunately. It does seem like igniter would be the “odd man out” so-to-speak here. Unlike language servers, its depended on by applications directly and interacted with mix tasks.

Why does Igniter need spitfire instead of the Elixir parser? Do you need error tolerance?

Also lib_elixir could extract the relevant files with an explicit step, and then check them into version control?