Sourceror is now available on hex.pm and supports Elixir versions down to 1.10

https://hexdocs.pm/sourceror/Sourceror.html

I finally got some time work on Sourceror, and there have been a bunch of bug fixes, and a new 0.9 version! Thanks to the folks that helped to test and iron out issues in both my comments syncer and Elixir 1.13 release candidate

There are some slight changes to the API, and more functionality is exposed. Also, the notebooks are now available as guide pages in hexdocs.

v0.9.0

1. Enhancements

• [Sourceror] to_string/2 now supports options for Code.quoted_to_algebra, like locals_without_parens
• [Sourceror] get_range/2 no longer considers comments when calculating the range. This can be enabled by passing the include_comments: true option
• [Sourceror.Patch] Introduced Sourceror.Patch with utilities to generate patches for the most common rewriting operations
• [Sourceror.Identifier] Sourceror.Identifier is now public

v0.8.x summary of bug fixes

• [Sourceror] Fixed comment spacing on binary operators
• [Sourceror] Take comment end of line counts into account to preserve spacing
• [Sourceror] Fixed an issue that caused comments in lists to be misplaced
• [Sourceror] Fixed issues that caused comments to be misplaced.
• [Sourceror] Updated internal normalizer to match latest Elixir 1.13 version.
• [Sourceror] Fixed an issue that caused newlines to be wrongly removed.
• [Sourceror] Fixed an issue that caused comments in pipelines to be misplaced.
• [Sourceror] Fixed issue that prevented keyword lists from preserving their
  original format in tuples.
• [Sourceror] get_range/1 now properly handles naked AST lists, like the ones
  coming from partial keyword lists, or stabs like a -> b.
• [Sourceror] get_range/1 now handles partial keyword list syntax instead of
  crashing.
• [Sourceror.Zipper] down/1 now correctly uses nil as the right siblings if
  the branch node has a single child.
• [Sourceror] Sourceror.get_range/1 now correctly calculates the range when
  there is a comment in the same line as the node.

The target audience is primarily tool authors, like elixir-ls or credo.

Yes, those are the kind of use cases I had in mind The Sourceror.to_string/2 function has an option to set the indentation level of the resulting code for that particular use case. I will probably add functions to know how many lines an ast node uses, so one could replace a line range instead of the whole file.

This started while exploring ways to allow credo to autofix some of the issues it finds, the multi alias expansion example derived from that.

Mostly finding what people find most cumbersome or confusing to do, I think the most important thing right now is to start experimenting. There may be some bugs in Code.quoted_to_algebra/2 too, some experiments in that front would be nice as well so we can add more regression tests to core Elixir

Sourceror 0.7.0 is out

This release adds a zipper API to improve the ergonomics of navigating and modifying the Elixir AST at will.

I added an introduction livebook to zippers: sourceror/zippers.livemd at main · doorgan/sourceror · GitHub

With this API, removing nodes or adding siblings is a straghtforward task in contrast with Macro.postwalk/Macro.prewalk. The livebook for the multi alias expansion demo was also updated with a new chapter that uses the zipper api to simplify the code.

It’s worth noting that it’s not a Sourceror specific implementation, it works for any Elixir AST. If you find yourself in the need of a zipper for your macros, Sourceror can help there too

Changelog:

1. Enhancements

• [Sourceror.Zipper] - Added a Zipper implementation for the Elixir AST based
  on Huet’s paper.

Sourceror 0.8.0 is out

This one is a bit small, just bug fixes and the addition of Sourceror.patch_string/2 which allows you to modify just some parts of a string, instead of having to print the whole tree. It receives the original string and a list of patches to be applied.

A patch is just a map with a :range pointing to the start and end positions to be replaced, and a :change that can be either a string, in which case the range will be replaced with it, or a function that accepts the original code in that range, and returns the string that replaces it.

This allows you to perform more fine grained changes to the source code, as you can use Sourceror.get_range/1 in combination with a traversal to generate the patches.

Also, now Sourceror.to_string/2 can receive a format: :splicing option that makes it easier to print elements of a keyword list without having to string the brackets yourself.

To illustrate these changes, here’s how a transformation for the Surface converter that renames the slot props: ... to args: ... would look like (from a discussion in the issue tracker):

Mix.install([{:sourceror, "~> 0.8"}])

code = """
defmodule Card do
  use Surface.Component

  slot footer
  slot header, props: [:item]
  slot default, required: true, props: [:item]
end
"""

{_, patches} =
  code
  |> Sourceror.parse_string!()
  |> Sourceror.postwalk([], fn
    {:slot, _, args} = quoted, state ->
      opts_node = Enum.at(args, 1, [])
      props_node = Enum.find(opts_node, &match?({{:__block__, _, [:props]}, _}, &1))

      if props_node do
        range = Sourceror.get_range(props_node)
        {{:__block__, meta, [:props]}, body} = props_node
        args_node = {{:__block__, meta, [:args]}, body}
        new_code = Sourceror.to_string([args_node], format: :splicing)
        patch = %{change: new_code, range: range}

        {quoted, %{state | acc: [patch | state.acc]}}
      else
        {quoted, state}
      end

    quoted, state ->
      {quoted, state}
  end)

code
|> Sourceror.patch_string(patches)
|> IO.puts

The next versions will be focused on exploring ways to make it easier to create the patches, and to make the apis more stable

Changelog:

1. Enhancements

• [Sourceror] Added Sourceror.patch_string/2
• [Sourceror] Added the format: :splicing option to Sourceror.to_string/2

2. Bug fixes

• [Sourceror] Now Sourceror.to_string/2 won’t produce invalid Elixir code when a keyword list element is at the beginning of a non-keyword list.
• [Sourceror] Now Sourceror.get_range/1 will take the leading comments into account when calculating the range.