halostatue

halostatue

Ecto Fragments and the PostgreSQL JSON `?` operator

How do I use the PostgreSQL JSON ? operator in Ecto? Assume that I have a table users that has a JSONB field data, and I want to test (in PostgreSQL) whether it has a key honorific.

The SQL for this is:

SELECT *
  FROM users
 WHERE data ? 'honorific';

Neither of these work:

from u in :users, where: fragment("? \? ?", u.data, "honorific")
from u in :users, where: fragment("? ? ?", u.data, "?", "honorific")

The former fails with a compile error (not enough parameters to match the number of question marks) and the latter fails in the query because it produces SELECT u0.* FROM users WHERE u.data '?' 'honorific'.

I think that this is a bug, and I’d be interested to know how we would need to use such operators, because PostgreSQL has several:

┌──────┬─────────┬───────────┬─────────────┬────────────────────────────┐
│ Name │ Left arg│ Right arg │ Result type │        Description         │
├──────┼─────────┼───────────┼─────────────┼────────────────────────────┤
│ <?>  │ abstime │ tinterval │ boolean     │ is contained by            │
│ ?    │ jsonb   │ text      │ boolean     │ key exists                 │
│ ?#   │ box     │ box       │ boolean     │ deprecated, use && instead │
│ ?#   │ line    │ box       │ boolean     │ intersect                  │
│ ?#   │ line    │ line      │ boolean     │ intersect                  │
│ ?#   │ lseg    │ box       │ boolean     │ intersect                  │
│ ?#   │ lseg    │ line      │ boolean     │ intersect                  │
│ ?#   │ lseg    │ lseg      │ boolean     │ intersect                  │
│ ?#   │ path    │ path      │ boolean     │ intersect                  │
│ ?&   │ jsonb   │ text[]    │ boolean     │ all keys exist             │
│ ?-   │ point   │ point     │ boolean     │ horizontally aligned       │
│ ?-   │         │ line      │ boolean     │ horizontal                 │
│ ?-   │         │ lseg      │ boolean     │ horizontal                 │
│ ?-|  │ line    │ line      │ boolean     │ perpendicular              │
│ ?-|  │ lseg    │ lseg      │ boolean     │ perpendicular              │
│ ?|   │ jsonb   │ text[]    │ boolean     │ any key exists             │
│ ?|   │ point   │ point     │ boolean     │ vertically aligned         │
│ ?|   │         │ line      │ boolean     │ vertical                   │
│ ?|   │         │ lseg      │ boolean     │ vertical                   │
│ ?||  │ line    │ line      │ boolean     │ parallel                   │
│ ?||  │ lseg    │ lseg      │ boolean     │ parallel                   │
└──────┴─────────┴───────────┴─────────────┴────────────────────────────┘

Marked As Solved

michalmuskala

michalmuskala

You can escape the question mark with \, but you need it doubled - once for string and once for fragment. This should work fine:

from u in :users, where: fragment("? \\? ?", u.data, "honorific")
16
Post #6

Also Liked

NobbZ

NobbZ

Instead of "? \\? ?" one can also use ~S"? \? ?", which removes one layer of escaping.

halostatue

halostatue

This does not answer the question I raised above, but I have found two workarounds for this using PostgreSQL. Both of these depend on an undocumented function (only the ? operator is documented) and a bit of digging to produce something similar as needed.

  1. Use the underlying function of ? directly. fragment("jsonb_exists(?, ?)", u.data, "honorific"). The functions for ?& and ?| are jsonb_exists_all and jsonb_exists_any, so those can be used as well (but note that the types are text[], so you would need to do fragment("jsonb_exists_any(?, ?)", u.data, ["honorific"])).

  2. Create a new operator analogous to ?, ?&, and ?| in your schema.

CREATE OPERATOR =~ (
  PROCEDURE=pg_catalog.jsonb_exists, LEFTARG=jsonb, RIGHTARG=text
);
CREATE OPERATOR =~& (
  PROCEDURE=pg_catalog.jsonb_exists_all, LEFTARG=jsonb, RIGHTARG=text[]
);
CREATE OPERATOR =~| (
  PROCEDURE=pg_catalog.jsonb_exists_any,LEFTARG=jsonb, RIGHTARG=text[]
);

Then you can construct these…

fragment("? =~ ?", u.data, "honorific")
fragment("? =~& ?", u.data, ["honorific"])
fragment("? =~| ?", u.data, ["honorific"])

Unfortunately, no one in the PostgreSQL community will understand these operators against json/jsonb types because you’ve created them, and you’re no longer writing standard PostgreSQL JSON queries. Unless you know these operators have been created, asking questions about these would cause confusion.

halostatue

halostatue

I should have tried that. This might be worth putting in the Ecto documentation.

halostatue

halostatue

Third post on this for those following along, how I figured all of this out. I’ve been playing in pg_catalog recently because I’ve been building some tooling to automate the construction of monthly table partitioning in PostgreSQL 9.6, so I’m as apt to dig into this now as ever.

If you need to figure out how to do this yourself, psql -E <database> is your friend, because when you run something like \doS (describe operators in all schemas), it will output a query like the one below (the one below is one I copied and modified to find all operators with a question mark in them for the table in the first post).

SELECT n.nspname as "Schema", o.oprname AS "Name",
       CASE WHEN o.oprkind = 'l' THEN NULL ELSE pg_catalog.format_type(o.oprleft, NULL) END AS "Left arg type",
       CASE WHEN o.oprkind = 'r' THEN NULL ELSE pg_catalog.format_type(o.oprright, NULL) END AS "Right arg type",
       pg_catalog.format_type(o.oprresult, NULL) AS "Result type",
       coalesce(pg_catalog.obj_description(o.oid, 'pg_operator'), pg_catalog.obj_description(o.oprcode, 'pg_proc')) AS "Description"
  FROM pg_catalog.pg_operator o
  LEFT JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace
 WHERE o.oprname OPERATOR(pg_catalog.~) '^([?])$'
   AND pg_catalog.pg_operator_is_visible(o.oid)
 ORDER BY 1, 2, 3, 4;

This will tell you the argument types for your aliased operator. In our case, left is jsonb and right is text. It still doesn’t tell you which function to use. For that, you need to look at pg_operator directly:

SELECT oprname, oprcode FROM pg_operator WHERE oprname = '?';

This gives us the answer that the ? operator uses jsonb_exists. If the function/procedure you’re using does not exist in the same schema as the operator you’re defining (it probably doesn’t in this case), you must fully qualify it to pg_catalog.jsonb_exists.

This tells us that our CREATE OPERATOR statement needs to look like:

CREATE OPERATOR =~ (
  PROCEDURE=pg_catalog.jsonb_exists, LEFTARG=jsonb, RIGHTARG=text
);

Be careful about overloading, as overloads require different types. You can qualify an operator by using the OPERATOR() function in a query (seen above in the introspection query):

fragment("? OPERATOR(pg_catalog.=~) ?", u.name, "Michal")

If you wish to drop an operator, you must know the arg types as well.

DROP OPERATOR =~(jsonb, text);
halostatue

halostatue

The need to query into the JSONB is relatively rare for my applications, and I don’t recall what made me start looking at this particular issue. Most of my needs are met by straight queries, and on the rare occasion I need to query into JSONB it’s better to just use fragment/1, which is why I was curious.

Where Next?

Popular in Questions Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement