ckampfe

ckampfe

Postgrex + Redshift. Anyone?

Hey folks,
Does anyone have experience using Postgrex with AWS Redshift, preferably recently and/or in production?
My team is looking to use Elixir to be a broker for a lot of our ETL data engineering work, and we make heavy use of Redshift, among other tools that traditionally have very good support for JVM languages.

I know there has been work done on this front before, but I’m unclear as to where it sits now. For example:

https://groups.google.com/forum/#!msg/elixir-lang-talk/QGB5Tcr18ms/-56rADBMEQAJ points to this fork that is at this point a little old and unloved: https://github.com/colinbankier/postgrex

as well as some stuff on the issue tracker for Postgrex: https://github.com/elixir-ecto/postgrex/issues?utf8=✓&q=redshift

Greatly appreciate any reports/help!

Most Liked

lackac

lackac

Since you are also using Redshift from Elixir you might be interested that we released RedshiftEcto (Redshift adapter for Ecto) yesterday. It aims to provide a good experience working with a Redshift database from Ecto. Features unsupported by Redshift will raise an error and the adapter supports table and column options specific to Redshift in migrations.

There’s a separate topic for the announcement:

talentdeficit

talentdeficit

Rebar3 Core Team

i use postgrex with redshift internally. you need version 0.13.1 or later (not 1.0.0-rc.0 or 1.0.0-rc.1 which predate the 0.13 series) but it should work fine

ckampfe

ckampfe

Hi @demem123,

The system is now in production and works well.

We’ve gone with using Ecto to run stringified/templated SQL against our Redshift cluster. The system is purely internal and only runs trusted queries, so we elected to go this route despite string interpolation in SQL being wildly insecure otherwise.

One thing working in our favor is that the queries we’re running against Redshift using this method are 1. largely administrative, 2. simple, 3. rarely change. For those reasons, we haven’t felt a great need to use Ecto’s query builder or anything on that level of sophistication.

Overall this works well enough for our particular workflow, but I would definitely not use this method if any piece of the system was public facing.

talentdeficit

talentdeficit

Rebar3 Core Team

redshift doesn’t support upsert and the redshift client library is actually just postgrex so if you try to use ecto upserts with redshift you’ll get sql errors in your queries

talentdeficit

talentdeficit

Rebar3 Core Team

postgrex has pooling built in. see the options for start_link

Where Next?

Popular in Questions Top

lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

Other popular topics Top

bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement