Fl4m3Ph03n1x
Code coverage tools for Elixir?
Background
I have a test suite and I need to know the coverage of the project.
I have played around with mix test --cover but I find the native erlang’s coverage analysis tool to be insufficient at best.
The native coverage tool doesn’t tell you about branch coverage nor function coverage. It’s only metric seems to be relevant lines which I have no idea how they calculate. For all I know, this is just the most basic form of test coverage: see if a given text line was executed.
What have you tried?
I have tried Coverex but the result was disastrous. Not only does it suffer from the same issues that the native tool does, it also seems not produce correct results as it counts imported modules as untested.
Or maybe it is doing a great job and my code is poorly tested, but I can’t know for sure because it doesn’t tell me how it is evaluating my code. Have 40% coverage in a file? What am I missing? I can’t know, the tool wont tell me.
I am now using ExCoveralls. It is considerably better than the previous options, it allows me to easily configure which folders I want to ignore, but it uses the native coverage tool, so it suffers pretty much from the same issues.
What do you want?
I was hoping to find something among the lines of Istanbul, or in this case nyc:
It’s test coverage analysis tells me everything I need to know, metrics and all:
Branches, Functions, Lines, Statements, everything you need to know is there.
Question
- Is there any tool that uses Istanbul for code coverage metrics with Elixir instead of the native erlang one?
- If not, is there a way to configure the native coverage tool to give me more information?
- Which metrics does the native coverage tool uses ?
Marked As Solved
hauleth
No exactly, the difference is that JS is interpreted, so it has no notion of “compiled code”. In Erlang on the other hand there is such possibility, and single line can result in multiple VM “statements”, for example:
Integer.to_string(a + 1)
Will result with 2 instructions:
{line,[{location,"lib/tasks.ex",6}]}.
{gc_bif,'+',{f,0},1,[{x,0},{integer,1}],{x,0}}.
{line,[{location,"lib/tasks.ex",6}]}.
{call_ext_only,1,{extfunc,erlang,integer_to_binary,1}}.
So you can see that it is hard to match statements to instructions, especially as in theory compiler is free to reorder commands as it pleases as long as the result is the same (this can be especially seen in C/C++ where it is written in standard, and result of it is that foo(a++, a++) is undefined behaviour).
Also Liked
cblavier
Just a brief update, because I needed to get solid code coverage for a decent phoenix app (umbrella app of 5 sub-apps, 17k loc).
On the app side, we’ve been using excoveralls : it is reliable, can be run in umbrella apps, with partitioned parallel testing and can output in different formats. 
Then we needed also a good UI to be able to drill down within our application code to know what to test to improve our code coverage. I tested:
-
coveralls.io: the UI is only OK but not really convenient nor pretty. I could not make Github Status checks or comments to work (in order to get instant code coverage feedback when opening a pull request). Integration in the CI was not really convenient as well because we had to disable partitioned testing to run all tests at once and upload a single report.
-
codeclimate.com was disapointing
Integration within CI was ok : we had to generate multiple json reports (one per partition) with excoverallsthen format them, merge them and upload the whole withcc_test_reporter(binary provided by code_climate). Github feedback is working well, but you can’t really explore your code source tree within codeclimate UI. No go
-
codecov.io is our choice. Integration within the CI is really simple : you can upload multiple report for a single build (one per test partition) and everything will be merged on the serverside. Github feedback is really insightful and the UI gives you the ability to drill down as you want to build your test coverage strategy
cblavier
Hey there, I just wrote this article:
hauleth
The point is that in Erlang there is no much difference between statements and lines, so I would roughly say that line = statement. Second stat is branch which also do not make much sense, as most of the time Erlang will use pattern match, so it will be split into 2 categories:
- function clauses
- lines
Both of these are available in cover module.
Functions are available as well as modules in cover, so that shouldn’t be much of the issue either.
So as you can see that metrics either do not make sense or are supported in Erlang’s cover.
dimitarvp
I have badly misread while I was on the phone. Apologies.
dimitarvp
Have you tried https://github.com/parroty/excoveralls?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"