anthonator

anthonator

Sharing a DBConnection pool with multiple Ecto Repo's

Is it possible to create a shared connection pool that can be used across many repos?

Most Liked

unnawut

unnawut

@anthonator Just dropping by to say that your approach inspired me to figure out something similar.

I did just slightly differently without a new connection pool module (and thus no delegations).

defmodule DB.SharedConnectionPool do
  alias DBConnection.ConnectionPool

  def child_spec({mod, opts}) do
    opts = Keyword.put_new(opts, :name, pool_name(opts))
    Supervisor.Spec.worker(__MODULE__, [{mod, opts}])
  end

  def start_link({mod, opts}) do
    case GenServer.start_link(ConnectionPool, {mod, opts}, start_opts(opts)) do
      {:ok, pid} -> {:ok, pid}
      {:error, {:already_started, pid}} -> {:ok, pid}
      error -> error
    end
  end

  defp pool_name(opts) do
    case Keyword.fetch(opts, :shared_pool_id) do
      {:ok, pool} -> String.to_atom("#{__MODULE__}-#{pool}")
      :error -> __MODULE__
    end
  end

  # Exact same as `DBConnection.ConnectionPool`
  defp start_opts(opts) do
    Keyword.take(opts, [:name, :spawn_opt])
  end
end

And I have the repos configured like this:

config :my_app, MyApp.Repo,
  pool: DB.SharedConnectionPool,
  shared_pool_id: :my_shared_pool_id

config :my_second_app, MySecondApp.Repo,
  pool: DB.SharedConnectionPool,
  shared_pool_id: :my_shared_pool_id

This way makes it also possible to specifically configure what repos should share a pool, and multiple shared pools can be created.

The downside is that it still relies on the connection info from the configs, which means if two repos with different configs share the same pool, there’s a race condition and one repo config will be taken to start the pool. But otherwise this approach has the least impact on how Ecto works.


Full code here: https://github.com/omisego/ewallet/pull/853 (disclaimer: I’m employed by this project)

dylan-chong

dylan-chong

With db_connection 2.4.0, i get this error

** (UndefinedFunctionError) function Solve.Utils.DB.SharedConnectionPool.checkout/3 is undefined or private
    (utils 0.1.0) Solve.Utils.DB.SharedConnectionPool.checkout(#PID<0.13239.0>, [#PID<0.13592.0>, #PID<0.13591.0>], [log: #Function<14.6198013/1 in Ecto.Adapters.SQL.with_log/3>, source: "query_history_v1", timeout: 15000, pool_size: 10, pool: Solve.Utils.DB.SharedConnectionPool])
    (db_connection 2.4.0) lib/db_connection.ex:1082: DBConnection.checkout/3
    (db_connection 2.4.0) lib/db_connection.ex:1407: DBConnection.run/6
    (db_connection 2.4.0) lib/db_connection.ex:574: DBConnection.parsed_prepare_execute/5
    (db_connection 2.4.0) lib/db_connection.ex:566: DBConnection.prepare_execute/4
    (ecto_sql 3.5.4) lib/ecto/adapters/sql.ex:690: Ecto.Adapters.SQL.execute!/4
    (ecto_sql 3.5.4) lib/ecto/adapters/sql.ex:682: Ecto.Adapters.SQL.execute/5
    (ecto 3.5.6) lib/ecto/repo/queryable.ex:224: Ecto.Repo.Queryable.execute/4
    (ecto 3.5.6) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (saved_query_service 0.1.0) lib/query_service.ex:55: Solve.SavedQueryService.fetch_history/2
    (query_runner 0.1.0) lib/query_runner/tidy_history_task.ex:7: Solve.QueryRunner.TidyHistory.run/0
    (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Adding:

  defdelegate checkout(pool, callers, opts), to: ConnectionPool

to the connection pool here Sharing a DBConnection pool with multiple Ecto Repo's - #5 by unnawut seems to fix the issue

mbuhot

mbuhot

This is the reason I just share a repo between all umbrella apps, and declare the schemas separately in each app. Otherwise you can easily max out the 200 connection limit on an AWS db.t2.small instance :smile:

anthonator

anthonator

This is typically what I do and I believe most other people using umbrella apps do as well. For me, the problem arises when using tools like Postgres schemas. You can have multiple logical datastores within the same physical database which share a connection limit. In that scenario each application should have it’s own Repo with a shared connection pool.

Pus, you can’t generate your migrations within the contextually relevant applications. That’s the real problem! Makes my eye twitch.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement