codeRaider

codeRaider

Build correctly Massive Data Structure

TL;DR
Better way and Efficient methods to build a massive data structure (like a snapshot), or this is job for another language?

In-deep
In a Redis server, we have segmented objects by parameters. The base object is institute, this have a base json object with parameters like address, phone number, code, id, etc. Then, this institute have nested keys with data (json object), like institute:<code>:teachers, institute:<code>:executive and others.
There is a builder that create this massive structure by many institutes with a list of institute codes, one by one, this generate a base structure and merge and add the nested objects.

A little example:

%{
    "institute-a" => %{
      address: "Street A 453",
      id: 12,
      status: "active",
      online: true,
      location: %{
        country: "United State",
        state: "CA",
        city: "Los Angeles"
      },
      teachers: %{
        "john-doe" => %{
          age: 35,
          asignatures: ["Art", "Music"],
          id: 999
        },
        ...
      },
      courses: %{
        "A1" => {
          students: %{
            "carol-doe" => %{
              age: 15,
              grades: %{
                "music" => "A",
                "arts" => "B+"
              },
              ....
            }
          }
        }
      }
    },
    ....
  }

The duration to build this structure was nice (5 secs), but more institutes was added to the Redis server, already we have 500 institutes.
I try many forms, but, when the data is so massive, the request time is like 15~20secs to show this request data or send as response in a channel event.
I know elixir is not friendly with this kind of process, this is temporary, but I need ideas to be more efficient and respecting the elixir way to code.

Most Liked

chulkilee

chulkilee

Did you find out what part is actually taking time mostly?

If building the data is 5 sec but the whole request takes 15-20… it means something takes 10-15s :wink:

Things to check before going further (e.g. rustler)

  • Use iolist not string for redis command (if the library supports)
  • Check Redis library/server setting which makes thing slow - especially redis response time (not your app response time)
  • Do not rebuild data again
  • Avoid copying large data across erlang nodes - maybe better to refetch from phx channel process
chulkilee

chulkilee

Okay then you have N+1 query problem - this is data fetching problem not necessarily elixir.

You may make it faster by performing N+1 queries in parallel (use Task for instance) - but in this case you loose the benefit of Redis pipelining.

Anyway you’re accessing the data not in “Redis way”. You may prefetch most values if knows the pattern in advance - e.g. using SCAN or it’s family.

Or you can use Lua script to preprocess a little bit on redis side.

As you have redis, you may use redis for caching as well - which is helpful if transformation is expensive compared to network bandwidth between erlang nodes and redis instance.

darnahsan

darnahsan

You can have a look at this it might be of some help or atleast give you idea over hwo to tackle your problem

chasers

chasers

I’m not sure it sounds like Flow may be something to look into for a short term solution. It should help spread the map process out across all your cores.

chasers

chasers

And yeah if you can cache the response for even like a second it may help quite a bit. Cachex provides a nice built in ttl feature that will automatically bust your cache after a set timeframe.

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement