I use this custom formatter for our CI to get the failures summarized at the bottom. There is a small race condition in it which can cause an extra line or two at the end, but it does the job. I’ve only ever used with along with --trace so not sure what it looks like with anything else (probably much the same).

defmodule MyApp.Test.Formatter do
  @moduledoc """
  Adds test failures to the bottom of the test output.

  This is mostly only useful when run with `--trace`.

  Usage:

     $ mix test --trace --formatter MyApp.Test.Formatter
  """

  use GenServer

  @format_cols 80

  @doc false
  def init(opts) do
    {:ok, pid} = GenServer.start_link(ExUnit.CLIFormatter, opts)
    {:ok, %{cli_formatter: pid, failures: []}}
  end

  def handle_cast({:test_finished, %{state: {:failed, errors}} = test} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    counter = length(state.failures) + 1

    error =
      ExUnit.Formatter.format_test_failure(test, errors, counter, @format_cols, &formatter/2)

    state = %{state | failures: [error | state.failures]}

    {:noreply, state}
  end

  def handle_cast({:suite_finished, _} = event, state) do
    GenServer.cast(state.cli_formatter, event)

    if Enum.any?(state.failures) do
      failures =
        state.failures
        |> Enum.reverse()
        |> Enum.join("\n")

      IO.puts("""

      #{String.duplicate("=", @format_cols)}

        Failures:

      #{failures}

      #{String.duplicate("=", @format_cols)}
      """)
    end

    {:noreply, state}
  end

  def handle_cast(event, state) do
    GenServer.cast(state.cli_formatter, event)

    {:noreply, state}
  end

  defp formatter(_key, value), do: value
end

I can’t answer your question directly – it’s been a long time since I last looked at test formatters – but one workaround is to just run mix test --failed afterwards, which will only run the tests that failed during the previous run.

If your concern is that you have too much test terminal output and that you’re finding it hard to track the output only of those that failed then maybe that command will help you.

Yes, that’s exactly my concern. Thanks for pointing me to mix test --failed. Unfortunately, it doesn’t help in case when failed test log is captured so you still have a lot of scrolling. Sometimes you just want to see the summary of failed tests - without investigating the log of the failed test.

I adapted this script a little to make it project-agnostic. It now fetches the manifest based on the Mix.Project.manifest_path()

# list_failed_tests.exs
manifest_path = Mix.Project.manifest_path() |> Path.join(".mix_test_failures")

if not File.exists?(manifest_path) do
  IO.puts("Can't find manifest file at #{manifest_path}")
  System.stop()
  Process.sleep(:infinity)
end

manifest_path
|> File.read!()
|> :erlang.binary_to_term()
|> elem(1)
|> Enum.group_by(fn {_, file} -> file end, fn {{_mod, name}, _file} -> name end)
|> Enum.each(fn {file, tests} ->
  IO.puts([
    IO.ANSI.red(),
    "Failures in #{file}:\n",
    Enum.map_join(tests, "\n", &"  * #{&1}"),
    "\n",
    IO.ANSI.reset()
  ])
end)

You can run it with MIX_ENV=test mix run list_failed_tests.exs

Hi there! I’ve same issue, that’s why I created simple library GitHub - balance-platform/ex_unit_summary: "Extension" for ExUnit

Feel free to use