jpinnix

jpinnix

Postgrex invalid byte sequence for encoding "UTF8": 0x00

Postgres is complaining that I am inserting invalid UTF8 into a text type:

(Postgrex.Error) ERROR 22021 (character_not_in_repertoire) invalid byte sequence for encoding "UTF8": 0x00

I checked the inserts with String.codepoints, and the only thing I found that was interesting was:

iex(2)> String.codepoints("1554942951-607301301834131-2-pdf_invoice_preview.png-original.png")
["1", "5", "5", "4", "9", "4", "2", "9", "5", "1", "-", "6", "0", "7", "3", "0",
 "1", "3", "0", "1", "8", "3", "4", "1", "3", "1", "-", "2", "-", "p", "d", "f",
 "_", "i", "n", "v", "o", "i", "c", "e", "_", "p", "r", "e", "v", "i", "e", "w",
 ".", "p", ...]

It truncates the ending “ng-original.png” Is that possibly the problem?

Full log:

INSERT INTO "documents" ("content_type","filename","new_filename","unique_id","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5,$6) RETURNING "id" ["image/png", "pdf_invoice_preview.png", "1554942951-607301301834131-2-pdf_invoice_preview.png-original.png", "1554942951-607301301834131-2", ~N[2019-04-11 00:35:51], ~N[2019-04-11 00:35:51]]
[debug] QUERY ERROR db=5.0ms
INSERT INTO "images" ("document_id","hash","uuid","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5) RETURNING "id" [12, <<38, 52, 51, 51, 51, 33, 37, 6, 0, 11, 122, 107, 171, 59, 158, 30>>, <<168, 246, 60, 242, 50, 132, 75, 81, 165, 27, 253, 122, 144, 194, 119, 156>>, ~N[2019-04-11 00:35:51], ~N[2019-04-11 00:35:51]]
[debug] QUERY OK db=0.2ms
rollback []
[info] Sent 500 in 173ms
[error] #PID<0.465.0> running DistanceWeb.Endpoint (connection #PID<0.464.0>, stream id 1) terminated
Server: localhost:4000 (http)
Request: POST /documents
** (exit) an exception was raised:
    ** (Postgrex.Error) ERROR 22021 (character_not_in_repertoire) invalid byte sequence for encoding "UTF8": 0x00
        (ecto_sql) lib/ecto/adapters/sql.ex:605: Ecto.Adapters.SQL.raise_sql_call_error/1
        (ecto) lib/ecto/repo/schema.ex:649: Ecto.Repo.Schema.apply/4
        (ecto) lib/ecto/repo/schema.ex:262: anonymous fn/15 in Ecto.Repo.Schema.do_insert/4
        (ecto) lib/ecto/multi.ex:579: Ecto.Multi.apply_operation/5
        (elixir) lib/enum.ex:1940: Enum."-reduce/3-lists^foldl/2-0-"/3
        (ecto) lib/ecto/multi.ex:563: anonymous fn/5 in Ecto.Multi.apply_operations/5
        (ecto_sql) lib/ecto/adapters/sql.ex:875: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
        (db_connection) lib/db_connection.ex:1415: DBConnection.run_transaction/4
        (ecto) lib/ecto/repo/transaction.ex:15: Ecto.Repo.Transaction.transaction/4
        (distance) lib/distance_web/controllers/document_controller.ex:24: DistanceWeb.DocumentController.create/2
        (distance) lib/distance_web/controllers/document_controller.ex:1: DistanceWeb.DocumentController.action/2
        (distance) lib/distance_web/controllers/document_controller.ex:1: DistanceWeb.DocumentController.phoenix_controller_pipeline/2
        (distance) lib/distance_web/endpoint.ex:1: DistanceWeb.Endpoint.instrument/4
        (phoenix) lib/phoenix/router.ex:275: Phoenix.Router.__call__/1
        (distance) lib/distance_web/endpoint.ex:1: DistanceWeb.Endpoint.plug_builder_call/2
        (distance) lib/plug/debugger.ex:122: DistanceWeb.Endpoint."call (overridable 3)"/2
        (distance) lib/distance_web/endpoint.ex:1: DistanceWeb.Endpoint.call/2
        (phoenix) lib/phoenix/endpoint/cowboy2_handler.ex:33: Phoenix.Endpoint.Cowboy2Handler.init/2

Marked As Solved

al2o3cr

al2o3cr

That’s the default behavior of IO.inspect; you can print the whole value with IO.inspect(some_value, limit: :infinity)

But the documents insert is followed by another, so I don’t think it’s the one that’s raising.

In the next insert, there’s a raw binary value for hash:

<<38, 52, 51, 51, 51, 33, 37, 6, 0, 11, 122, 107, 171, 59, 158, 30>>

which has a zero byte. What’s the type of the images.hash column?

Also Liked

jpinnix

jpinnix

images.hash was type :string (varchar in Postgres)

Had to change it to :binary (bytea in Postgres)

Thanks @al2o3cr!!

Dado

Dado

If you are having this issue: ERROR 22021 (character_not_in_repertoire) invalid byte sequence for encoding “UTF8”, my problem was that on the migration for that table was wrong, I forgot the refences(:table_name, opts)
Example:
BEFORE: add :question_id, :string
AFTER: add :question_id, references(:questions, on_delete: :nothing, type: :binary_id)

Where Next?

Popular in Questions Top

lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New

We're in Beta

About us Mission Statement