ckampfe
Postgrex + Redshift. Anyone?
Hey folks,
Does anyone have experience using Postgrex with AWS Redshift, preferably recently and/or in production?
My team is looking to use Elixir to be a broker for a lot of our ETL data engineering work, and we make heavy use of Redshift, among other tools that traditionally have very good support for JVM languages.
I know there has been work done on this front before, but I’m unclear as to where it sits now. For example:
https://groups.google.com/forum/#!msg/elixir-lang-talk/QGB5Tcr18ms/-56rADBMEQAJ points to this fork that is at this point a little old and unloved: https://github.com/colinbankier/postgrex
as well as some stuff on the issue tracker for Postgrex: https://github.com/elixir-ecto/postgrex/issues?utf8=✓&q=redshift
Greatly appreciate any reports/help!
Most Liked
lackac
Since you are also using Redshift from Elixir you might be interested that we released RedshiftEcto (Redshift adapter for Ecto) yesterday. It aims to provide a good experience working with a Redshift database from Ecto. Features unsupported by Redshift will raise an error and the adapter supports table and column options specific to Redshift in migrations.
There’s a separate topic for the announcement:
talentdeficit
i use postgrex with redshift internally. you need version 0.13.1 or later (not 1.0.0-rc.0 or 1.0.0-rc.1 which predate the 0.13 series) but it should work fine
ckampfe
Hi @demem123,
The system is now in production and works well.
We’ve gone with using Ecto to run stringified/templated SQL against our Redshift cluster. The system is purely internal and only runs trusted queries, so we elected to go this route despite string interpolation in SQL being wildly insecure otherwise.
One thing working in our favor is that the queries we’re running against Redshift using this method are 1. largely administrative, 2. simple, 3. rarely change. For those reasons, we haven’t felt a great need to use Ecto’s query builder or anything on that level of sophistication.
Overall this works well enough for our particular workflow, but I would definitely not use this method if any piece of the system was public facing.
talentdeficit
redshift doesn’t support upsert and the redshift client library is actually just postgrex so if you try to use ecto upserts with redshift you’ll get sql errors in your queries
talentdeficit
postgrex has pooling built in. see the options for start_link








