vadimshvetsov

vadimshvetsov

How to properly sanitize multiple ts_vector fields with reusable macro?

I’ve crawled couple related topics but still can’t implement reusable module for full-text search. I’m stuck with this module and can’t understand why it can’t find any row even when it has to. Looks like there is interpolation problem with creating multiple ts_vector but I’m not sure.

My goals:

  1. to abstract later search for other entities and have this api API
  2. to speed up this search with creating indexes while it has some joins (is it possible?) And is it normal to search within related tables?
  3. Consider to add tsv field with triggering updates and creating indexes and having to_tsvector value in db.
defmodule MyApp.Performer.Search do
  import Ecto.Query

  @fields ~w(bio first_name last_name phone email city)
  @nullable_fields ~w(name)

  defmacro tsquery(fields, terms, language \\ "english") do
    quote do
      fragment(
        "to_tsvector(?, ?) @@ to_tsquery(?, ?)",
        unquote(language),
        unquote(fields),
        unquote(language),
        unquote(terms)
      )
    end
  end

  defmacro tsrankcd(fields, terms, language \\ "english") do
    quote do
      fragment(
        "ts_rank_cd(to_tsvector(?, ?), to_tsquery(?, ?))",
        unquote(language),
        unquote(fields),
        unquote(language),
        unquote(terms)
      )
    end
  end

  def run(query, search_term) do
    search_term = normalize(search_term)
    fields = fields_to_tsvector_fields(@fields, @nullable_fields)

    queryable =
      from q in query,
        join: p in assoc(q, :profile),
        left_join: t in assoc(q, :topics),
        where: tsquery(^fields, ^search_term),
        order_by: [
          desc: tsrankcd(^fields, ^search_term)
        ],
        distinct: [q.id]
  end

  defp normalize(terms, operator \\ "&") do
    terms
    |> String.downcase()
    |> String.trim()
    |> String.replace(~r/\s+/u, " #{operator} ")
    |> (&(&1 <> ":*")).()
  end

  def fields_to_tsvector_fields(non_nullable_fields, nullable_fields) do
    nullable_fields
    |> Enum.map(fn field -> "coalesce(#{field}, ' ')" end)
    |> (&(non_nullable_fields ++ &1)).()
    |> Enum.join(" || ' ' || ")
  end
end

Would highly appreciate any advice or suggestion, thanks.

Marked As Solved

thiagomajesk

thiagomajesk

Hi @vadimshvetsov! At work, we have implemented a small lib that helps us abstract what we need for full-text search. Take a look at our repo and see if it helps: https://github.com/raidcorp/searchy.

Also Liked

vadimshvetsov

vadimshvetsov

Thanks a lot for helping me.

I’ve starred https://github.com/raidcorp/searchy and will keep an eye on it. The reason why I’ve done this on my own is because I need to rank results and search in related tables.

So I’ve ended up with this one:

At first I’ve added pg_term extension for incomplete word search:

defmodule MyApp.Repo.Migrations.AddPgTrgmExtension do
  @moduledoc """
  Create postgres pg_trgm extension and indices
  """

  use Ecto.Migration

  def up do
    execute("CREATE EXTENSION pg_trgm")
  end

  def down do
    execute("DROP EXTENSION pg_trgm")
  end
end

Then migrated searchable tables and add tsvector field, trigger and index:

defmodule MyApp.Repo.Migrations.AddPerformersSearch do
  use Ecto.Migration

  def up do
    alter table("performers") do
      add :tsvector, :tsvector
    end

    create index(:performers, [:tsvector], using: "GIN")

    execute("""
    CREATE OR REPLACE FUNCTION performers_tsvector_trigger()
    RETURNS trigger AS $$
    begin
      new.tsvector := setweight(to_tsvector('russian', coalesce(new.bio, '')), 'A');

      return new;
    end
    $$ LANGUAGE plpgsql;
    """)

    execute("""
    CREATE TRIGGER performers_tsvector_update
    BEFORE INSERT OR UPDATE ON performers
    FOR EACH ROW EXECUTE PROCEDURE performers_tsvector_trigger();
    """)

    alter table("users") do
      add :tsvector, :tsvector
    end

    create index(:users, [:tsvector], using: "GIN")

    execute("""
    CREATE OR REPLACE FUNCTION users_tsvector_trigger()
    RETURNS trigger AS $$
    begin
      new.tsvector := setweight(to_tsvector('russian', coalesce(new.last_name, '')), 'A') ||
      setweight(to_tsvector('russian', coalesce(new.first_name, '')), 'B') ||
      setweight(to_tsvector('russian', coalesce(new.email, '')), 'C') ||
      setweight(to_tsvector('russian', coalesce(new.phone, '')), 'C') ||
      setweight(to_tsvector('russian', coalesce(new.city, '')), 'D');

      return new;
    end
    $$ LANGUAGE plpgsql;
    """)

    execute("""
    CREATE TRIGGER users_tsvector_update
    BEFORE INSERT OR UPDATE ON users
    FOR EACH ROW EXECUTE PROCEDURE users_tsvector_trigger();
    """)

    alter table("topics") do
      add :tsvector, :tsvector
    end

    create index(:topics, [:tsvector], using: "GIN")

    execute("""
    CREATE OR REPLACE FUNCTION topics_tsvector_trigger()
    RETURNS trigger AS $$
    begin
      new.tsvector := setweight(to_tsvector('russian', coalesce(new.name, '')), 'A');

      return new;
    end
    $$ LANGUAGE plpgsql;
    """)

    execute("""
    CREATE TRIGGER topics_tsvector_update
    BEFORE INSERT OR UPDATE ON topics
    FOR EACH ROW EXECUTE PROCEDURE topics_tsvector_trigger();
    """)
  end

  def down do
    execute("DROP TRIGGER performers_tsvector_update on performers;")
    execute("DROP FUNCTION performers_tsvector_trigger();")

    alter table("performers") do
      remove :tsvector
    end

    drop index("performers", [:tsvector])

    execute("DROP TRIGGER users_tsvector_update on performers;")
    execute("DROP FUNCTION users_tsvector_trigger();")

    alter table("users") do
      remove :tsvector
    end

    drop index("users", [:tsvector])

    execute("DROP TRIGGER topics_tsvector_update on performers;")
    execute("DROP FUNCTION topics_tsvector_trigger();")

    alter table("topics") do
      remove :tsvector
    end

    drop index("topics", [:tsvector])
  end
end

Added tsvector type taken from https://github.com/raidcorp/searchy source:

defmodule MyApp.Ecto.Types.TSVector do
  use Ecto.Type

  def type, do: :tsvector

  def cast(tsvector), do: {:ok, tsvector}

  def load(tsvector), do: {:ok, tsvector}

  def dump(tsvector), do: {:ok, tsvector}

  def embed_as(_), do: :self

  def equal?(term1, term2), do: term1 == term2
end

Added tsvector field to all searchable ecto schemas:

    field :tsvector, Proling.Ecto.Types.TSVector

Added Search.Helpers module for convience:

defmodule MyApp.Search.Helpers do
  defmacro tsquery(tsvector, terms, language \\ "russian") do
    quote do
      fragment(
        "? @@ to_tsquery(?, ?)",
        unquote(tsvector),
        unquote(language),
        unquote(terms)
      )
    end
  end

  defmacro tsrankcd(tsvector, terms, language \\ "russian") do
    quote do
      fragment(
        "ts_rank_cd(?, to_tsquery(?, ?))",
        unquote(tsvector),
        unquote(language),
        unquote(terms)
      )
    end
  end
end

And finally added API for using in context:

defmodule MyApp.Production.Performer.Search do
  import Ecto.Query
  import MyApp.Search.Helpers

  @spec search(Ecto.Query.t(), any()) :: Ecto.Query.t()
  def search(query, search_term) do
    search_term = normalize(search_term)

    from q in query,
      join: p in assoc(q, :profile),
      left_join: t in assoc(q, :topics),
      where:
        tsquery(p.tsvector, ^search_term) or tsquery(q.tsvector, ^search_term) or
          tsquery(t.tsvector, ^search_term),
      order_by: [
        desc: tsrankcd(p.tsvector, ^search_term),
        desc: tsrankcd(q.tsvector, ^search_term),
        desc: tsrankcd(t.tsvector, ^search_term)
      ],
      distinct: [q.id]
  end

  defp normalize(terms, operator \\ "&") do
    terms
    |> String.downcase()
    |> String.trim()
    |> String.replace(~r/\(|\)\[|\]\{|\}/u, "")
    |> String.replace(~r/\s+/u, " #{operator} ")
    |> (&(&1 <> ":*")).()
  end
end

I’m gonna star @thiagomajesk answer with searchy because it`s source code greatly led me to the final destination. Also I would like to mention this awesome answer - Search Bar Feature (Phoenix Searching)

Hope this step-by-step example could help someone to get things done.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Ecto does not interpolate any values at all, ever. Rather it uses SQL parameters to avoid SQL injection. If you turn on debug logging you should be able to see the query that Ecto runs. Try running that in PSQL and tweaking the values until you get what you want, then adjust your ecto query accordingly.

thiagomajesk

thiagomajesk

I don’t know if I understood exactly what you need but as long as you have a ts_vector field generated for the table you want to search, I guess you are good to go. Our use-case is very simplistic and the lib itself is just a thin wrapper that translates the queries according to the postgres docs. If this is not exactly what you need or it’s incomplete, feel free to open a discussion on the repo so we can talk about it :blush:.

cpgo

cpgo

I think it depends on how much complexity you want to put on the search optimization.
The query you posted might work, or you could create a materialized view with your tsvectors.
As most things, I would benchmark it before commiting to a more complex solution.

fuelen

fuelen

I think you can avoid all that complexity with fields_to_tsvector_fields function, @fields and @nullable_fields just by using concat_ws.

defmacro concat_ws(first_value, second_value, separator) do
  quote do
    fragment("concat_ws(?, ?, ?)", unquote(separator), unquote(first_value), unquote(second_value))
  end
end

separator was moved to the last argument to have an ability to compose concat_ws calls via pipe operator

ts_query(
  p.bio
  |> concat_ws(p.first_name, " ")
  |> concat_ws(p.last_name, " ")
  |> concat_ws(p.phone, " ")
  |> concat_ws(p.email, " ")
  |> concat_ws(p.city, " ")
  |> concat_ws(t.name, " "),
  ^search_term
)

Where Next?

Popular in Questions Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement