fedme

fedme

How to investigate DB errors like "client timed out because it queued and checked out the connection for longer than..."

Hi,

We’re starting to collect many of these errors in our logs and I am trying to understand how we can investigate them further:

Postgrex.Protocol (#PID<0.1264.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.1555.0> timed out because it queued and checked out the connection for longer than 15000ms

I have read this topic and I understand these are caused by slow queries.
Now I am wondering, can Postgrex/Ecto be configured to log the slow query together with the above error?

I know I can find slow queries in a different way (e.g. with Postgres’pg_stat_statements extension), but it would be extremely useful if I could get some indication of what DB operation caused the timeout detailed in the error log.

Is that error log about a DB connection pool process crashing due to a timeout? Will it cause the caller process to crash as well? Maybe there’s a way to trap the timeout and log it somehow?

Thanks for your help!

Marked As Solved

LostKobrakai

LostKobrakai

Not sure you understood my comments correctly. You can get results by looking for slow queries, but a pool timeout is not guaranteed to come from a slow query. It could just as much be from a lot of fast queries accumulating their time needed to complete – or to an extreme by not running any query at all, but just leaving the connection checked out for longer than the timeout.

The better way to diagnose this is to see if the stacktrace printed can be extended to get out of general code paths and into ones which are useful for debugging.

Also Liked

pdgonzalez872

pdgonzalez872

Moving away from Elixir here, you could also enable logs on Postgres and monitor them as you play around with the suggestions above. I have some notes about this here. Please post your findings!

LostKobrakai

LostKobrakai

Yes and yes. But given this is a timeout for the pooled connection this means it’s not bound to a particular query. You can e.g. do Repo.checkout(fn -> Process.sleep(:infinity) end) and you’d get that error without any query involved.

c4710n

c4710n

IMO, you have to diagnose this problem from the slow query.

Edit: checkout the below explaination of @LostKobrakai.


If you guess that the root cause is slow queries. You can find the slow queries by using the tools provided by PostgreSQL (which you said above). Or, you can use Ecto directly. Here’s how:

Ecto supports metrics which are reported by :telemetry.
You can log slow queries like this - Output slow Ecto queries to logs · GitHub.

(The gist is picked randomly from the Internet…

LostKobrakai

LostKobrakai

IEx has some special ways to deal with exceptions. Generally the caller crashes.

justincy

justincy

I’m also interested in logging the associated query, if there is one. I thought about adding it to the Postgrex.Protocol DBConnection.ConnectionError error message that gets printed but I haven’t been able to figure out where that’s being generated.

The error which gets returned by ecto is

{:error,
 %DBConnection.ConnectionError{
   message: "connection is closed because of an error, disconnect or timeout",
   severity: :error,
   reason: :error
 }}

Unfortunately at that point we are no longer sure if it’s a timeout anymore (it’s just a generic connection error) which makes it less useful to just wrap all Repo methods, though I might still have to do that.

Where Next?

Popular in Questions Top

yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement