You can use the instrument — runtime_tools v2.3 module to get more information on the memory stored. This kind of tracking is enabled by default for the binary_* and driver_alloc. Given this is sqlite it’s likely some NIF related data. I also seem to remember that driver_alloc is about NIF memory.

Not really an answer to your question but I was curious where those values in the chart actually come from. It appears to be this call to :erlang.system_info({:allocator_sizes, _}).

Still not clear to me what the values actually mean, though. The docs for system_info/1 imply it’s intentionally undocumented.

On the problem page, what kind of work is it doing? It might give us some more insight.

Oh never mind, here they are. Which one is spiking in the chart? Those colors are unfortunately not very distinguishable.

I’ve been able to figure out how to use instrument. For future references. Run a script like this to get the best information out: elixir --erl “+Muatags true” sqlite_reproduction.exs

So I ran :instrument.allocations before the query and after the query, and thse are the only significant changes that I see:

image666×201 11.5 KB

So this line:nif_internal: {1999, 438, 40, 10, 10, 9, 15474, 0, 0, 10, 10, 1, 0, 0, 0,
0, 0, 0}

Running with exqlite gives me no such thing:

For completeness here is the full script that I used:

Mix.install([
    {:ecto_sql, "~> 3.13.2"},
    {:ecto_sqlite3, "~> 0.22"},
    {:exqlite, "~> 0.33.1"}
])

Application.put_env(:myapp, Repo, database: "./dev.db")

defmodule Repo do
  use Ecto.Repo,
    otp_app: :myapp,
    adapter: Ecto.Adapters.SQLite3
end

defmodule Main do
  @sql ~s"""
  SELECT
    s0."id",
    e1."id",
    e1."type",
    e1."data",
    e1."inserted_at",
    s0."stream_id",
    s0."stream_version"
  FROM "stream_events" AS s0
  INNER JOIN "events" AS e1 ON s0."event_id" = e1."id"
  WHERE s0."stream_id" = '$all' AND s0."stream_version" >= 0
  ORDER BY s0."id" DESC
  LIMIT 1
  """

  def sqlite do
    IO.inspect "before"
    print_memory()
    save_allocations("./before_sqlite")


    {:ok, _} = Repo.start_link([])
    _s1 = Ecto.Adapters.SQL.query!(Repo, @sql, nil)

    IO.inspect "after"
    print_memory()
    save_allocations("./after_sqlite")
  end

  def exqlite do
    IO.inspect "before"
    print_memory()
    save_allocations("./before_exqlite")

    {:ok, conn} = Exqlite.Sqlite3.open("./dev.db")
    {:ok, statement} = Exqlite.Sqlite3.prepare(conn, @sql)
    {:row, _results} = Exqlite.Sqlite3.step(conn, statement)
    IO.inspect "after"
    print_memory()
    save_allocations("./after_exqlite")
  end


  defp print_memory() do
    IO.inspect "memory: #{:erlang.memory(:total) / 1_000_000}"
  end

  defp save_allocations(name) do
    {:ok, x} = :instrument.allocations
    File.write!(name, inspect(x, limit: :infinity, pretty: true))
  end
end

# Main.sqlite()
Main.exqlite()