What this library does is near identical to what I manually do in most of my projects

It’s not hard at all to do manually, though this library does make the pattern more obvious.

Glad to know this is similar to a pattern you might already be using. To your point, the entire library/helper is only about 20 LOC and can pretty easily be done inline. The value to me is definitely more about documenting a consistent pattern, a clean external API, and not recreating the wheel (even if easily recreated) in every context.

I suspect a lot of experienced Phoenix devs already have some sort of helper they are using that serves the same type of purpose.

I’ve now moved one project fully to this alternate style and it feels pretty good thus far. It is certainly more verbose in the controller, but very easy to see what is going on.

In the controller…

featured_projects =
  ProjectMgmt.list_projects([
    &ProjectMgmt.order_projects_by_name/1,
    &ProjectMgmt.preload_project_photos/1,
    &ProjectMgmt.preload_project_categories/1,
    &ProjectMgmt.filter_featured_projects/1,
    &ProjectMgmt.filter_published_projects/1
  ])

As opposed to how it is using TokenOperator…

featured_projects =
  ProjectMgmt.list_projects(include: [:photos, :categories], filter: [:featured, :published])

Clearly it is much cleaner in the controller with TokenOperator. Additionally, the concept of “defaults” is built in. Thus, I would have pre-configured list_projects to sort by name as an overridable default.

This alternative route loses the built in defaults, so order_projects_by_name is passed explicitly, but I don’t think that’s so bad. And there is no question of what order these queries will be run.

For more complex queries involving scoping based upon user permissions, I think this alternate method is easier to follow. Suppose I want to scope the rooms that a user can reserve. I was using the following with TokenOperator in the controller…

Inventory.list_rooms_for(location,
  scope: {user, :reserve}
)

That eventually calls a scope function within my context, but there are some gymnastics involved where it isn’t immediately obvious how it gets there.

This alternate method is more obvious in the controller…

Calendar.list_rooms([
  &Calendar.order_rooms_by_name/1,
  &Calendar.scope_rooms(&1, user, :reserve))
])

In practice, I’m hiding that ugly bit of code to run those queries off in a utility:

defmodule Utilities.QueryRunner do
  def run(query, additional_queries) when is_list(additional_queries) do
    Enum.reduce(additional_queries, query, fn additional_query, query ->
      additional_query.(query)
    end)
  end

  def run(query, additional_query) do
    run(query, [additional_query])
  end
end

This is then referenced within the context:

alias Utilities.QueryRunner

def list_projects(queries \\ []) do
  Project
  |> QueryRunner.run(queries)
  |> Repo.all()
end

As of now I’m leaning toward this simplified method, but both have their advantages.

@mathieuprog I think using something like QueryBuilder is a valid way to do it and that does some cool stuff in figuring out whether to preload/join, query associated columns, etc. It is just a different take on it.

I originally started down the path of something similar, but ended up with the more abstract TokenOperator (and now tossing around the idea of an alternative method without a package at all) with the following goals:

1. No dependency on an ORM. Unopinionated about whether this is a query or multi or whatever token.
2. Unopinionated about the language used for interacting with a context (e.g. include, preload, join, paginate, etc.). Use the words that make sense for your app.
3. Consistency of usage. Do the same thing for the least AND most complex queries. Write them out in the context and then reference them via atoms in the controller. No matter how powerful QueryBuilder (or any opinionated Ecto wrapper) becomes, there are still going to be times when you need to do something more complex than its API supports. You can keep adding features, but at some point it might become so big and complex that it might be easier to just write the queries in the context since Ecto has a great API already. While verbose, they are generally easy to write and I often need those queries anyway for operations within the context.
4. Flexibility - Ability to not have to need a list_posts and list_published_posts and list_published_featured_posts for every little tweak to a query. This is probably the core thing that TokenOperator, QueryBuilder, and this alternate method all do so it barely needs mentioning here.
5. Transparency - Easy to see (and change) the queries that are being run. The alternate method I’m talking about here uses direct references to context functions, which doesn’t get any more obvious or transparent. TokenOperator uses atoms and still makes it easy to get at the functions, but with a layer of abstraction that makes it slightly less so. As @LostKobrakai mentioned, passing atoms like in TokenOperator and QueryBuilder allows the context to dictate what queries are made. That has been useful but, in practice, has provided me less value and transparency than I thought it might, which is why I brought up this alternate method.
6. Also probably goes without saying, but all of these methods are trying to avoid directly interacting with Ecto, which is what contexts are pushing us to do.

As an aside, I considered writing an opinionated Ecto “adapter” for TokenOperator as an example of how to use it. It is made to be the core of higher-level abstractions like QueryBuilder.

Anyway, this is a nuanced thing and I’m trying to tease out some of the differences between the approaches, but I think they are all useful. I’m clearly still undecided on how I want to do it so I appreciate things like QueryBuilder being around as an option.

By token, I just mean a data structure that can be passed around and operated upon. Two common examples of tokens are an Ecto query and Plug.Conn. This post explains it much better than I can… https://rrrene.org/2018/03/26/flow-elixir-using-plug-like-token/

TokenOperator is actually pretty abstracted in that it doesn’t care what type of token you are passing around or what naming you use for the options. I chose an obvious use-case to explain how it can be used. I don’t think I did a particularly good job explaining what it is though.

I started out calling it MaybeQuery, dependent upon Ecto, and with hard-coded opinionated option names like filter, order_by, and preload. That would have been easier to explain, but less flexible. Instead, this allows you to configure your own API conventions and naming and use it beyond that example use case if one presents itself.