ccgintranet

ccgintranet

Inserting Binary, Repo & Ecto

Hello everyone,

First post here and I am hoping someone can help expand my knowledge as I am clearly not handling a situation we are having properly.

We are using ExCrypto which directs to the underlying :crypto.block_encrypt/decrypt.

Our Ecto schema looks something like this:

    schema "bank" do
      field :uuid, :binary
      field :iv, :binary
      field :tag, :binary
    end

Part of the logic calls to ExCrypto.rand_bytes which returns a random binary stream.

An example here is if I attempt to insert it using Repo I will receive an expected struct that allows me to decrypt some cipher text.

%Api.Bank{
  __meta__: #Ecto.Schema.Metadata<:loaded, "bank">,
  id: 2,
  iv: <<155, 6, 136, 177, 177, 49, 143, 88, 129, 95, 129, 232, 197, 222, 16,
    200>>,
  tag: <<76, 5, 248, 99, 14, 234, 17, 201, 220, 233, 231, 86, 228, 193, 77,
    102>>,
  uuid: "a2eQg6kJiQ82CdpQD5SjBA=="
}

By requerying the database with that particular uuid (this would be a random string) I seem to get a completely different output.

%Api.Crypt{
  __meta__: #Ecto.Schema.Metadata<:loaded, "bank">,
  id: 2,
  iv: <<63, 6, 63, 63, 63, 49, 63, 88, 63, 95, 63, 63, 63, 63, 16>>,
  tag: <<76, 5, 63, 99, 14, 63, 17, 63, 63, 63, 63, 86, 63, 63, 77, 102>>,
  uuid: "a2eQg6kJiQ82CdpQD5SjBA=="
}

This tells me that the data being passed over to Repo changed somewhere between that call and it going into the database.

Working from this dataset in memory is fine, using ExCrypto.encrypt & ExCrypto.decrypt but as soon as I persist it to the database and retrieve it its garbled.

The reason its being persisted is we are using a backend RabbitMQ queue which picks up something we are doing at a later date.

My question, is am I not understanding how binaries work in Elixir or is something happening to my raw binary down the line when Repo inserts it?

Base.encode64 and Base.decode64 works fine, nothing is lost to and fro the database which makes me question how is the binary stream being handled here.

We use a MySQL database (5.6), InnoDB, utf8mb4_0900_ai_ci using BLOB columns.

Help is much appreciated,

Many thanks

Most Liked

voughtdq

voughtdq

It seems like PHP’s openssl_encrypt automatically encodes to base64. I wonder if you would have the same problem when using the OPENSSL_RAW_DATA option in your PHP implementation.

You may need to create a custom ecto type to encode/decode the binary.

voughtdq

voughtdq

<<155, 6, 136, 177, 177, 49, 143, 88, 129, 95, 129, 232, 197, 222, 16, 200>>
<<63, 6, 63, 63, 63, 49, 63, 88, 63, 95, 63, 63, 63, 63, 16>>

Something is changing it. If the byte is greater than 128 (with the current data we have), it seems to get dropped to 63.

NobbZ

NobbZ

128 and greater is not valid ASCII anymore, nore were the given string valid UTF8.
63 is ASCII (and UTF8) for a questionmark.

Can you please check the generated database query? Does that already contain the garbled data or is the query generated correctly? If you try to insert an opposing dataset manually into the database, containing only a single byte greater than 127, can you retrieve it back correctly?

Where Next?

Popular in Questions Top

yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

We're in Beta

About us Mission Statement