Fl4m3Ph03n1x

How to have excoveralls consider total testing coverage from all apps in umbrella project?

Background

I am using excoveralls in an umbrella project with several children.

I have set up the children to use Excoveralls like this:

For normal Elixir applications:

 def project do
    [
      app: :app1,
      version: "3.1.0",
      build_path: "../../_build",
      config_path: "../../config/config.exs",
      deps_path: "../../deps",
      lockfile: "../../mix.lock",
      elixir: "~> 1.16",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      test_coverage: [tool: ExCoveralls],
      preferred_cli_env: preferred_cli_env()
    ]
  end

And for a Phoenix child, the configuration is slightly different:

def project do
    [
      app: :phoenix_app_1,
      version: "2.2.1",
      build_path: "../../_build",
      config_path: "../../config/config.exs",
      deps_path: "../../deps",
      lockfile: "../../mix.lock",
      elixir: "~> 1.16",
      elixirc_paths: elixirc_paths(Mix.env()),
      compilers: Mix.compilers(),
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      test_coverage: [tool: ExCoveralls],
      preferred_cli_env: preferred_cli_env()
    ]
  end

 defp deps do
    [
      # Phoenix
      # ....

      # I dont know why I have to explicity include excoveralls here
      # but if I dont `mix coveralls` wont work for Phoenix child apps
      {:excoveralls, "~> 0.18", only: :test},
    ]
  end

For both types of apps the preferred_cli_env set to this:

defp preferred_cli_env,
    do: [
      coveralls: :test,
      "coveralls.detail": :test,
      "coveralls.post": :test,
      "coveralls.html": :test,
      "coveralls.github": :test
    ]

Problem

When I run mix coveralls it runs all tests from each child app sequentially, due to my customized mix test alias:

defp aliases do
    child_tests =
      Path.wildcard("apps/*")
      |> Enum.map(&String.replace(&1, "apps/", ""))
      |> Enum.map(fn app -> "cmd --app #{app} mix test --color" end)

    [test: child_tests]
  end

So what ends up happening, is that coveralls will run the coverage for each app, and then only present me with the coverage for the last app run.

This is a problem. I don’t want the coverage of only the last child app, I want the coverage of all child apps, inside the project, so i know the overall test coverage of the project.

Questions

Is this possible to do with coveralls?
If so, how?

11 comments

#umbrella #test #excoveralls

7 850 11

2024-01-15 11:42:53 UTC

Marked As Solved

LostKobrakai

Named processes are a common pattern for sure. The issue is also not the named process by itself. The issue is the hardcoded dependencies on those names (and their backing process) into your codebase, which means tests can no longer opt out of using that named process in favor of another instance.

As soon as you look into applications we would generally consider to be libraries you see the pattern of keying a whole tree of processes to a given name all the time. E.g. in ecto processes are keyed to the repo name, in broadway the processes are keyed to the name of the broadway pipeline. E.g. see here how the MyBroadway name determines the names of all child processes: Broadway — Broadway v1.0.7

Post #8

Also Liked

LostKobrakai

Tbh that sounds like you’re manually starting applications, which you generally shouldn’t need to be doing in the first place. If you have your dependencies setup correctly between your umbrella applications (using {:app, in_umbrella: true}) all necessary applications should start up no matter what mix task you run and if it’s for the whole umbrella or just a single app within.

Post #4

LostKobrakai

Fair. You got global state in your application and your tests show you that this is not necessarily the best design.

My general suggestion for global state is refactoring those singleton processes (with a fixed name) to processes, which can be given a name. That way you can start multiple concurrent instances of those processes with different names.

Given that you can start a separate set of processes per test. You can use dependency injection (of various forms) to inject the name (or names) to be used by your business logic when interacting with those processes.

Production can use a single name (or names), which effectively gives you singleton behaviour in production.

The issues you run into are why I generally suggest that test should either not depend on state present on processes started by applications or at least that state should be keyed to the running test. That approach is essentially how the ecto sandbox or mox allow you to run tests concurrently.

While this is talking about processes the same can equally be applied to other global resources.

Post #6

LostKobrakai

Usually, but I tend to mirror the order of parameters GenServer.call has as well. It’s imo the more reasonable order if you happen to pass both the server and item.

Post #12