joselo

joselo

Shopify feature: Full text search or joins to find text in multiple tables

Hi every one.

I’m building a kind of Shopify clone in Phoenix, and I need to implement a navigation menu, so the idea is find any occurrence in the Link section and find the text in different schemas or tables, this is a full-text search, I read that I can do using PostgreSQL creating a materialized view, etc. or the other solution could be just build an Ecto query with joins.

My question is:

Do you know which could be a good approach to build the query for this feature?. As you can see in this example there could be 4 “schemas”: Product, Collection, Pages.

If anyone has been experience building something likes this or have a good approach to build the query, I really appreciate your help.

Btw: I would like to avoid to use ElastiSearch, since Postgres has a good features to support indexing (if there is some approach with indexing).

Thanks!

Marked As Solved

amnu3387

amnu3387

You need to do a couple of things in order to have this working, you could probably do joins but I personally prefer the materialised view approach with a background process that just refreshes it. This is viable if you don’t need the search to be reflected immediately as long as you can create a unique index on the table and Postgresql will be able to refresh the view concurrently.

The process can be summarised as:

  1. Create the materialised view
def change do
    execute("""
    CREATE MATERIALIZED VIEW your_search_table AS
    SELECT 'product' AS result_type, products.id, products.title AS display_text, to_tsvector('english', products.title) AS tsv_search, product_categories.category_id AS category_id, product_subcategories.subcategory_id AS subcategory_id FROM products
    LEFT JOIN product_categories ON product_categories.product_id = products.id
    LEFT JOIN product_subcategories ON product_subcategories.product_id = products.id
    WHERE products.published
    UNION ALL
    SELECT 'store' AS result_type, store.id, stores.company_name AS display_text, to_tsvector('english', stores.company_name) AS tsv_search, NULL AS category_id, NULL AS subcategory_id FROM stores WHERE stores.type = 'marketplace'
    """)

    execute("CREATE UNIQUE INDEX id_type_index ON your_search_table (result_type, id);")
    execute("CREATE INDEX tsv_index ON your_search_table USING gin(tsv_search);")
    execute("CREATE INDEX categories_index ON your_search_table (category_id, subcategory_id) WHERE result_type = 'product'")
end

Notice this had things removed like setting weights on the tsv’s so it might not work in copy paste but should give you an idea.
Perhaps you should also add a reverse action to the execute statement. The unique index is necessary for the concurrent refresh. By using a type and an id where each id is unique in the scope of the type you can be sure that index is unique. If you can’t for some reason have that then you would create new materialised views on the fly, and rename & drop them as you go, so you have always a table ready and no queries get blocked waiting for the refresh.

  1. define a schema for it:
defmodule SearchTable do
  use Ecto.Schema

  @derive {Jason.Encoder, only: [:result_type, :id, :display_text]}
  @primary_key false
  schema("your_search_table") do
    field :result_type,                     :string, primary_key: true
    field :id,                                     :id, primary_key: true
    field :display_text,                    :string
    field :tsv_search,                      TSVector.Type
    field :category_id,                    :id
    field :subcategory_id,              :id
  end

end
  1. Define a type for the TSVector:
defmodule TSVector.Type do
  @behaviour Ecto.Type

  def type, do: :tsvector
  def cast(tsvector), do: {:ok, tsvector}
  def load(tsvector), do: {:ok, tsvector}
  def dump(tsvector), do: {:ok, tsvector}
end
  1. Now to query the table, you’ll need to have a fragment to make it usable in normal queries, so defining one somewhere and importing it into whatever context queries:
defmacro tsquery(field, text) do
    quote do
      fragment("?::tsvector @@ to_tsquery('english', ?)", unquote(field), unquote(text))
    end
end
  1. You’ll also need to prepare the text for the tsquery, when it comes in:
def split_text_for_tsquery(text) do
    String.split(text, " ", trim: true)
    |> Enum.reject(fn(text) -> Regex.match?(~r/\(|\)\[|\]\{|\}/, text) end)
    |> Enum.map(fn(token) ->  token <> ":*" end)
    |> Enum.intersperse(" & ")
    |> Enum.join
end

Then you can use it as:

def search(terms, params) do
    case split_text_for_tsquery(terms) do
      "" -> []
      search ->
        YourSearchTable
        |> where([yst], tsquery(yst.tsv_search, ^search))
        |> Db.Repo.all
    end
end
  1. Finally you’ll need to keep the table updated. You can write a process that loops itself every X and just refreshes it, like every 5min.
 Db.Repo.query("REFRESH MATERIALIZED VIEW CONCURRENTLY #{@your_search_table_name}", [], [timeout: 360_000])

Also Liked

joselo

joselo

Awesome, thanks so much for such a complete answer, I will start with this approach. :sunglasses:

Where Next?

Popular in Questions Top

belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
beno
I will often find my self writing things similar to: case some_value do nil -&gt; something() "" -&gt; something() _ -&gt; someth...
New

We're in Beta

About us Mission Statement