bigfish1
Best way to handle ETL tasks in Elixir?
I am looking for suggestions on the best way to handle ETL tasks in elixir or if I should just do these tasks in SQL + dbt + airflow.
I’m building an application around forecasting the effects of training on athletes to be used by team athletic staff. The users will upload data from wearable devices and we extract what we need then have two stages of transformation before writing the DB and the user can see these graphed on a nice dashboard.
I currently have these transformations written in SQL and can orchestrate them with dbt, but I would like to simplify this.
Would gen stage be a good fit here or should I avoid putting this pipeline into the application and keep it separate. Ideally I would like to move away from SQL and do this all with Nx Explorer because the cost of containers running is a lot cheaper then managed databases.
If anyone has any experience or advice here it would be greatly appreciated. Thank you in advance
Marked As Solved
benwilson512
GenStage / Broadway and other libraries in that family are a good fit when you have a sort of continuous stream of data to operate on. If when you say “users will upload data” you mean that data is being continuously streamed to you then GenStage could be a good fit to at least get the data written, and then run queries to do further transformations.
However if by “users will upload data” you mean it’s more of a bulk / batch operation then I’d consider something like Oban to do this in a “job” structure. A file gets uploaded and you kick off a job to go ingest data from the file, and run further queries to transform that data. It depends a bit on how data comes to you.
Also Liked
benwilson512
Gotcha. I mean doing this in memory isn’t impossible by any stretch, but it may be cheaper to use something like clickhouse that is optimized for these sorts of bulk queries / transformations. We’ve been testing it lately for internal reporting and IoT data management and it is crazy fast, while still giving you a lot of the flexibility you’d expect from a SQL interface.
EDIT: @dimitarvp I mean explorer is rust if you’re using the polars backend. The issue is the same either way: If in memory is fine, do rust / polars / explorer / whatever but if not, an OLAP database is probably the next best bet.
bigfish1
Thanks, this will be more of batch data. When players do an activity the wearable device records this as a file. After completion these get uploaded manually or pushed to a webhook. Thanks for all your help!
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Enabling companies to succeed by building software people love.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"