Should Ecto be chucked out altogether for data science workflows?

IMO. If you can use Explorer for data science, then that would be my choice, because it was designed for this use case. Ecto was designed for business/application logic (and many operations are structured with this goal in mind).

So the issue is, let’s say you have a universe of 500 000 possible securities which is I believe a lower bound on what the bloomberg terminal is capable of showing. Each security has tons of metadata associated with it. Also, for say, a stock index, or a yield curve, many tickers must be joined up in “groups”. So as you can see we’re starting to have transaction-like OLTP workflows here in addition to the ticker storage columnar store.

We have:
ticker_timeseries_data - many_to_one - ticker
ticker - many_to_many - ticker_groups

We may also, when we use NX in a “live” condition (streaming inward data), have analyses that must be performed live too. So we need to keep track of those

analylsis - many_to_many - ticker
analysis - many-to_many - ticker_group

This looks a lot like “orders - customers” or “users - roles” for which Ecto is great. But it’s not great for columnar data. And the exact converse for ADBC.

So I basically have to have two database abstractions in my app. This is because unlike Python and R, which own “static” data science, the big pitch IMO for Elixir is “live” data science, with streaming data. But that’s a lot more complex situation in many cases than “static” where it’s often one-off style workflows. Live is inevitably going to have more structure to the application which is where Ecto can come in. Also, Phoenix Liveview is miles ahead of anything (except maybe ObservableHQ) in this regard.

Given the whole Elixir / BEAM stack’s way-better story on live applications than PYthon or R, I’m ensuring the bloomberg stuff I’m writing is geared that way too (in addition to “batch” history). Rblpapi for example has no concept of “live” subscription data, and in Python it’s hard.

Anyway I thought it was worthwhile elucidating my thoughts on this. I’ll for now go with Ecto + ADBC.