Maxximiliann
Using concurrency to quickly collate data from multiple sources
Given-
Vehicles is a list of 50 VIN numbers (vin_number).
Colors, Makes, Models, Transmissions, Fuel_economy, Horsepower and Torques are each a list of 50 VIN numbers with their respective features.
For instance:
Colors = [%{vin_number: 5YJSA1DG9DFP14705, color: Black} ...]
Makes = [%{vin_number: 5YJSA1DG9DFP14705, make: DB8 GT} ...]
get_colors, get_makes, get_models, get_transmissions, get_fuel_economy, get_horsepower and get_torques are functions which get the respective colors, makes, models, etc., etc., for a particular vin_number.
Example:
def get_colors(vin_number) do
Enum.find(Colors, &(&1).vin_number == vin_number)
|> Map.get(:color)
end
Collating all of this data into a new map, super_cars, by vin_number-
def super_cars do
Enum.map(vehicles, &
%{
vin: (&1).vin_number,
color: get_colors((&1).vin_number)
make: get_makes((&1).vin_number)
model: get_models((&1).vin_number)
transmission: get_transmissions((&1).vin_number)
fuel_economy: get_fuel_economy((&1).vin_number)
horsepower: get_horsepwer((&1).vin_number)
torque: get_torques((&1).vin_number)
}
)
end
So here’s my question:
How can Task.async be utilized to optimize the time it takes to create the new super_cars map? (Is there perhaps a better approach? Ecto, maybe?)
As always, thanks for your generous and patient insights 
Most Liked
al2o3cr
This isn’t directly relevant to your question about using parallelism, but if you’re concerned about performance consider converting your lists into maps:
colors = [%{vin_number: 5YJSA1DG9DFP14705, color: Black} ...]
map_colors = Map.new(colors, fn c -> {c.vin_number, c} end)
Then a function like get_colors is a map lookup, not a linear search.
chrisjowen
No worries, and although I am not convinced you will need such things for this (I could be wrong just not enough info) I think its worth pointing out your options if you do need to look at parallel processing in Elixir
Firstly all abstractions including Task.async all live on top of the core process model of beam, and its really worth your time understanding this fully.
The next stage is to understand about GenServers and how they encapsulate generic process behaviour (https://hexdocs.pm/elixir/GenServer.html)
After this you may want to still use Task.async or maybe https://hexdocs.pm/elixir/Task.html#async_stream/3
If this is not enough for you then the excellent GenStage (https://hexdocs.pm/gen_stage/GenStage.html) gives you some real control when producing/consuming large datasets.
Finally, there are interesting abstractions above GenStage such as:
Basically there are a lot of ways to do concurrent data processing in Elixir
chrisjowen
Maybe, as mentioned there are other overheads in concurrency. If you only have 50 records the question is how many records would each async task process. If you process say 1 item per task it may work out slower than the single process call.
The only way to tell is to try this with different configurations. My gut is that you would be better off keeping this as a single process call and doing smaller optimisations like I mentioned ( @al2o3cr just showed what I mean in their answer, keying by vin number will reduce your lookup time).
LostKobrakai
Depending on how you source the data you might be able to parallelize data gathering by putting the source data into an ETS table with read concurrency. So you don’t need to copy the source data, but parallelize the querying part.
chasers
Yeah I mean I assume these are all in a database somewhere so I’d focus on making a process for each attribute which independently caches those locally in ETS periodically. And then just lookup the vin in ETS when you need it. If not ETS even just consolidating them in a single Postgres table…
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"