Hi everyone,

I didn’t want to “spam” the form with release announcements after my initial post about benchee back in June but I figured with all the changes and new features now might be a good time to write something again! If it’s too much, please tell me

I just released new versions of my benchmarking library benchee along with benchee_csv, also introducing new formatters benchee_json and finally benchee_html to create nice HTML reports, with 4 different graphs that can also be exported as PNG images! And of course, there also is a blog post about all of it.

benchee has come a long way and I’m particularly excited about it supporting running your suite with different inputs as different implementations may behave differently depending on input size or structure. Also I changed the API of the main interface after a short but good discussion in this very forum. And I gotta say it looks way more elixir now

alias Benchee.Formatters.{Console, HTML}
map_fun = fn(i) -> i + 1 end
inputs = %{
  "Small (10 Thousand)"    => Enum.to_list(1..10_000),
  "Middle (100 Thousand)" => Enum.to_list(1..100_000),
  "Big (1 Million)"       => Enum.to_list(1..1_000_000),
}

Benchee.run %{
  "tail-recursive" =>
    fn(list) -> MyMap.map_tco(list, map_fun) end,
  "stdlib map" =>
    fn(list) -> Enum.map(list, map_fun) end,
  "body-recursive" =>
    fn(list) -> MyMap.map_body(list, map_fun) end,
  "tail-rec arg-order" =>
    fn(list) -> MyMap.map_tco_arg_order(list, map_fun) end
}, time: 10, warmup: 10, inputs: inputs,
   formatters: [&Console.output/1, &HTML.output/1],
   html: [file: "bench/output/tco_detailed.html"]

This then produces outputs thanks to the HTML formatter as you can see in this example report or get a preview with this image:

ips1222×849 50.8 KB

So yeah, I hope you like it. Would be great to hear what you like, or even better what you are missing, not liking or bugs so that I can improve and extend benchee and its associated libraries

Thanks!
Tobi

So here is what was promised! Benchee 0.99 and 1.0 (along with all plugins) are a reality

There aren’t many super hardcore new features, except for showing the absolute difference in the average of measurements to help arguments & informed decisions. Otherwise it’s some polish, documentation, house keeping etc.

0.99 introduces a bunch of deprecation warnings, when you eliminate them you should be good to upgrade to 1.0.

Changelog
Blog Post

I also want to thank everyone here for their input etc

Happy benchmarking

Hello friends, long time no see. Enjoy a new Benchee version with a bugfix (finally!), reduction counting and profiler after benchmarking (yay!) as well as some musings on Open Source and why it took so long.

And 0.14.0 is finally live for your pleasure! The major feature is drastically increased precision when measuring: from microseconds to nanoseconds! There’s a blog post showing off the most important changes but you can also check out the Changelog with all the changes.

I used Benchee couple times, and it’s definitely a solid solution - probably my “go to” one for all my benchmarking needs.

That said, one thing that bothers me each time I look at the README is that the example benchmark is not inside a module. Code that is not inside modules, is not compiled, but interpreted. This gives vastly different performance characteristics and makes benchmarks pretty much useless.
This is not a huge problem in the example, since the functions immediately call a module (so only the initial, anonymous function call is interpreted), but can lead to false results with more complex things.