I wrote a short X thread about my solution. It’s loosely based on my favourite llm eval tool promptfoo.

I’m a bit in a rush so I’m just leaving a link to the thread, plz don’t bash me for that

https://x.com/jskalc/status/1920455423972311400

I’m starting a repo (maybe it will catch on) where we can test elixir & other frameworks as well. Open to collaboration and happy for folks to tear it apart: GitHub - ash-project/evals: Tools for evaluating models against Elixir code, helping us find what works and what doesn't

In this case it’s not even meant to be used as a library. I want to make it a central tool we can run to evaluate LLM assistants skill with Elixir and its package ecosystem.

I did yes I was halfway done with mine when I saw yours and ultimately I wanted it to be focused on using a raw data format (yaml) so that it could be indexed and used for many purposes etc, so I plowed on. Perhaps my stuff could be replaced with your impl, but I wanted to get the data going and worry about the details after/let folks submit PRs. The repo I shared is not made to help people eval their solutions, but to be a central tool for the ecosystem.

Let’s address your questions one by one

1. Project is doing pretty good, still iterating on basically everything
2. ExUnit is great, just there were some challenges to solve:

• ensure I won’t run “normal” unit tests hitting remote APIs
• We’re using Req for making LLM calls, as described here. When tests are running, I’m adding a custom Req step caching responses on the disk. That way our bills won’t go out of control.
• failure / success of a test is useful, but not enough to iterate. We needed a way to understand what exactly is being sent to the LLM and what is the response to fix it. Sometimes there are multiple messages. A custom ExUnit reporter handles it. Here’s the code (not adjusted at all but should be enough to get an idea). You make request in any way you want, and then send it to reporter to be included in the output: Postline.TestReporter.report_llm_call(TestModule, request, response). TestModule is needed because there might be multiple tests running at the same time, we need to know to which test attribute given LLM call.

image1200×760 50.3 KB

• having evals integrated with the codebase is immensely powerful. It let’s us test not only LLMs but also a context pipeline, eg:

  test "Add what Zelensky said about respect during the meeting", %{post: post} do
    post = apply_scenario(post, :cont_trump_zelensky)
    post = add_message(post, user("Add what Zelensky said about respect during the meeting"))
    message = get_completion(post)
    assert_tool "set_editor_content", message
    assert_llm "What Zelensky said about respect during the meeting has been added in the post.", message
  end

• assert_llm is just a simple prompt, based on promptfoo. You can find it in the previous gist.

3. The challenge with ExUnit is in it’s synchronous nature - we often want to declare multiple tests in a single module, but they run synchronously, even with async: true. This is fine for fast tests but for complex LLM cases - not really. Also risk of running “regular” ExUnit tests is there. So I definitely see a place for something very similar to ExUnit, but with slightly different parallelism and built-in reporting capabilities.

4. Yes. My not-technical co-founder wrote most of prompts and evals, with help from cursor

@zachdaniel I think it might be interesting for you as well. I really like expressiveness of Elixir code for evals - sometimes you want to run a chain of LLM calls, sometimes your asserts are complex, and this can’t really be covered with YAML rules. Promptfoo tries, but honestly it’s quite messy. They even provided an escape hatch to write assertions in JS.

All in all, I really like an idea of ex_eval. Just, personally I’d still go with my approach instead of defining asserts / rules in inflexible structs