toraritte

toraritte

Why can't `:timestamptz` be set up as default timestamp for migrations in config?

Hi,

Trying to make :utc_datetime the global default for Ecto schemas and migrations in our app, and use timestamptz as default datetime type in PostgreSQL. Was able to figure these out, but there seems to be inconsistency around migration configurations - and because I don’t understand what’s happening, having a hard time letting it go.

Schemas

Making :utc_datetime the default at most can only be done on a module-per-module basis, but at least it’s straightforward.

defmodule ANV.Accounts.User do

  use Ecto.Schema

  # EITHER
  @timestamps_opts [type: :utc_datetime]

  schema "users" do

    field :username, :string

    # OR
    timestamps(type: :utc_datetime)
  end
end

Migrations

In migration: timestamps(type: :timestamptz)

Found the simplest solution in Time zones in PostgreSQL, Elixir and Phoenix by @hubertlepicki :

create table(:events) do
  add :title, :string

  timestamps(type: :timestamptz)
end

Which makes total sense, because Ecto.Migration.timestamps/1 will forward the type info to Ecto.Migration.add/3.

This solution also seems to automatically choose the right Elixir struct, which is DateTime, instead of the default NaiveDateTime, because if the corresponding schema timestamps are not set to :utc_datetime, any DB operations will fail.

Application-wide: migration_timestamps

From the Ecto.Migration docs:

  • :migration_timestamps - By default, Ecto uses the :naive_datetime type, but you can configure it via:

    config :app, App.Repo, migration_timestamps: [type: :utc_datetime]
    

Naively assumed that exchanging :utc_datetime to :timestamptz and simply using timestamps() in the migration would do the trick (based on how Ecto.Migration.timestamps/1 is implemented), but got an error message instead (see below).

It also seems superfluous to use a global config with :utc_datetime, because one would have to provide :timestamptz in the migration anyway, simply overwriting the global setting.

I assume that providing the type directly to timestamps/1 goes through the Postgrex adapter, setting the correct Elixir type, but using config has a side effect somewhere along the line.

** (DBConnection.EncodeError) Postgrex expected %DateTime{}, got ~N[2019-10-01 17:45:34]. Please make sure the value you are passing matches the definition in your table or in your query or convert the value accordingly.
    (postgrex) lib/postgrex/type_module.ex:897: Postgrex.DefaultTypes.encode_params/3
    (postgrex) lib/postgrex/query.ex:75: DBConnection.Query.Postgrex.Query.encode/3
    (db_connection) lib/db_connection.ex:1148: DBConnection.encode/5
    (db_connection) lib/db_connection.ex:1246: DBConnection.run_prepare_execute/5
    (db_connection) lib/db_connection.ex:1342: DBConnection.run/6
    (db_connection) lib/db_connection.ex:540: DBConnection.parsed_prepare_execute/5
    (db_connection) lib/db_connection.ex:533: DBConnection.prepare_execute/4
    (postgrex) lib/postgrex.ex:198: Postgrex.query/4
    (ecto_sql) lib/ecto/adapters/sql.ex:658: Ecto.Adapters.SQL.struct/10
    (ecto) lib/ecto/repo/schema.ex:649: Ecto.Repo.Schema.apply/4
    (ecto) lib/ecto/repo/schema.ex:262: anonymous fn/15 in Ecto.Repo.Schema.do_insert/4
    priv/repo/seeds.exs:21: anonymous fn/2 in :elixir_compiler_1.__FILE__/1
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
    priv/repo/seeds.exs:16: (file)
    (elixir) lib/code.ex:813: Code.require_file/2

In the end, I don’t mind the extra typing, and timestamps(type: :timestamptz) explicitly conveys type in every migration, but still curious what I am missing.

Thanks!

Most Liked

josevalim

josevalim

Creator of Elixir

Naively assumed that exchanging :utc_datetime to :timestamptz and simply using timestamps() in the migration would do the trick (based on how Ecto.Migration.timestamps/1 is implemented), but got an error message instead (see below).

This should work. However, if you update your migrations, you must update your schema. So a combination of:

config :app, App.Repo, migration_timestamps: [type: :timestamptz]

And @timestamps_opts [type: :utc_datetime] is enough for everything.

Ecto does not automatically figure out which type to use from the migration/database, exactly because a single database type may be one or more Elixir types. Case in point, utc_datetime can be used for both timestamp and timestamptz. The way timestamptz works is that the DB converts to UTC. The way utc_datetime works is that Elixir’s adapter converts to UTC before writing to the database. So since utc_datetime already guarantees the data will be normalized, you don’t strictly need timestamptz.

15
Post #2
champeric

champeric

I’ll chime in, because this is a subject that always annoyed me about how it was handled by default and I need to manually change things to my taste (I’m using PostgreSQL).

First, I want my timestamps to be timestamptz in the DB. Forget that it’s badly named (confusion about timezone), it is the correct type to represent a “point in time”. If your DB uses timestamp (without tz), it’s ambiguous and someone needing to interface with the DB outside your app won’t have the slightest clue what they represent (is it local time (ouch…) or UTC?). Using the correct type makes confusion impossible.

Anyway, most of the things I change have been mentioned in this thread.

Make sure migration timestamps are timestamptz by default and have a default, which makes it easier to insert rows in DB manually for testing. It’s important that the default timezone is set to UTC (it could be set in the DB, but I prefer to have my system work either way. It’s important because Ecto was doing some casting where where: token.inserted_at > ago(^days, "day") was converted to SQL inserted_at > $2::timestamp + (interval ...) and the only this conversion works correctly is if the default timezone is UTC.

config :my_app, MyApp.Repo,
  migration_primary_key: [type: :identity],
  migration_timestamps: [type: :timestamptz, default: {:fragment, "now()"}],
  after_connect: {Postgrex, :query!, ["SET TimeZone TO 'UTC';", []]}

As already mentioned, I have a custom schema:

defmodule MyApp.Schema do
  defmacro __using__(_) do
    quote do
      use Ecto.Schema
      @timestamps_opts [type: :utc_datetime_usec]
    end
  end
end

I don’t use the generators much, but still I have a private copy of the phx.gen.schema templates.
schema.ex, make it use my schema:

defmodule <%= inspect schema.module %> do
  use <%= hd(Module.split(schema.repo)) %>.Schema

migration.exs, make sure fields with utc_datetime and utc_datetime_usec are converted correctly to timestamptz, and at the same time make sure text is always used because I can’t stand the sight of varchar(255):

<%
internal_type_mapper = fn
  :string -> :text
  :utc_datetime_usec -> :timestamptz
  other -> other
end

type_mapper = fn
  :utc_datetime -> ":timestamptz, size: 0"
  {:array, :utc_datetime} -> "{:array, :timestamptz}, size: 0"
  {:array, subtype} -> inspect({:array, internal_type_mapper.(subtype)})
  other -> inspect(internal_type_mapper.(other))
end
%>defmodule <%= inspect schema.repo %>.Migrations.Create<%= Macro.camelize(schema.table) %> do
...
... <%= type_mapper.(Mix.Phoenix.Schema.type_for_migration(v)) %> ...
toraritte

toraritte

I was also wrong about this one, it is in plain sight in the docs:
https://hexdocs.pm/ecto/Ecto.Schema.html#module-schema-attributes

# Define a module to be used as base
defmodule MyApp.Schema do
  defmacro __using__(_) do
    quote do
      use Ecto.Schema
      @primary_key {:id, :binary_id, autogenerate: true}
      @foreign_key_type :binary_id
    end
  end
end

# Now use MyApp.Schema to define new schemas
defmodule MyApp.Comment do
  use MyApp.Schema

  schema "comments" do
    belongs_to :post, MyApp.Post
  end
end
LostKobrakai

LostKobrakai

config :my_app, MyApp.Repo,
  migration_timestamps: [type: :utc_datetime_usec]

defmodule MyApp.Ecto.Schema do
  defmacro __using__(_) do
    quote do
      use Ecto.Schema
      @timestamps_opts [type: :utc_datetime_usec]
    end
  end
end

That’s what I usually do.

It can only be reiterated that timestamptz despite its name does not store timezones though. The timestamp is stored in UTC. Though it does make timezone conversions within queries a bit less verbose with the db knowing it’s utc.

It also enables automatic conversion to/from the session timezone for postgres clients - which I find more confusing than helpful given different clients can see different values. Ecto does use the binary protocol of postgres, which doesn’t do sessions anyways.

LostKobrakai

LostKobrakai

It depends. The database never sees any information about the struct. All it sees is the value (as mapped to by Ecto.Type.dump/2) you try to write. If the value is still valid for the db column there’s nothing to complain. If that’s not the case you’ll get a database error at runtime. So mismatches between different timestamp values are unlikely to cause issues, as it the end the value written to the db is always a timestamp.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement