josevalim

josevalim

Creator of Elixir

Proposal: ICU/Unicode calendar/datetime formatting

NOTE: this is a focused thread, so we appreciate if everybody stayed on topic. Feel free to comment anything in regards to calendar formatting but avoid off-topic or loosely related topics. For example, if you would like to discuss or propose other Calendar/DateTime features, please use a separate thread.

Hi everyone,

This is a proposal for calendar/datetime formatting in Elixir. The formatting will use the markup specified Unicode’s Locale Date Markup Language.

Here is how the API will look like:

Calendar.format(date_or_time_or_datetime, "yyyy-MM-dd HH:mm:ss.SSSSSS")
#=> {:ok, "2018-11-29 13:19:41.032412"}

The Calendar.format/2 entry point accepts any calendar type, using structural typing. This means we will be able to format any map that has the fields being formatted.

We will also support a third argument, which is a formatter module, that can be used for translation of month names, eras, etc:

Calendar.format(date_or_time_or_datetime, "yyyy-MM-dd HH:mm:ss.SSSSSS", MyApp.PTBR)

Would you also be able to proxy all of this to Gettext or CLDR if you want to:

Calendar.format(date_or_time_or_datetime, "yyyy-MM-dd HH:mm:ss.SSSSSS", Gettext.Calendar)

In other words, the third argument supports translation but not localization. For example, CLDR specifies the short and long formats for dates and times that each region/locale uses but we won’t support those as we believe those can be trivially built on top of Elixir:

defmodule CLDR do
  def long_date(date_or_time_or_datetime) do
    Calendar.format(date_or_time_or_datetime, CLDR.Locale.long_date, CLDR.Locale)
  end
end

Note: this proposal was written by José Valim and Michał Muskała.

The ICU syntax

As described in the ICU page:

A date pattern is a string of characters, where specific strings of characters are replaced with date and time data from a calendar when formatting.

The Date Field Symbol Table below contains the characters used in patterns to show the appropriate formats for a given locale, such as yyyy for the year. Characters may be used multiple times. For example, if y is used for the year, ‘yy’ might produce ‘99’, whereas ‘yyyy’ produces ‘1999’. For most numerical fields, the number of characters specifies the field width. For example, if h is the hour, ‘h’ might produce ‘5’, but ‘hh’ produces ‘05’. For some characters, the count specifies whether an abbreviated or full form should be used, but may have other choices.

Two single quotes represents a literal single quote, either inside or outside single quotes. Text within single quotes is not interpreted in any way (except for two adjacent single quotes). Otherwise all ASCII letter from a to z and A to Z are reserved as syntax characters, and require quoting if they are to represent literal characters.

“Stand Alone” values refer to those designed to stand on their own, as opposed to being with other formatted values. “2nd quarter” would use the stand alone format (QQQQ), whereas “2nd quarter 2007” would use the regular format (qqqq yyyy).

The complete specification can be found here. Elixir will implement a subset of those formats, outlined below.

Format Description Examples Source
G abbreviated_era AD; BC Calendar.year_of_era/1 + Formatter.abbreviated_era/1
GG wide_era Anno Domini; Before Christ Calendar.year_of_era/1 + Formatter.wide_era/1
GGG narrow_era A; B Calendar.year_of_era/1 + Formatter.narrow_era/1
u+ year 2004 struct.year
yy two_digits_year_of_era 4, 14, 14, 14 Calendar.year_of_era/1
y+ year_of_era 4, 14, 214, 2014 Calendar.year_of_era/1
D+ day_of_year 189 Calendar.day_of_year/3
M, MM month 1, 01 struct.month
MMM abbreviated_month Nov struct.month + Formatter.abbreviated_month/1
MMMM wide_month November struct.month + Formatter.wide_month/1
MMMMM narrow_month N struct.month + Formatter.narrow_month/1
d+ day 1, 14, 31 struct.day
Q, QQ quarter 2, 02 Calendar.quarter_of_year/3
QQQ abbreviated_quarter Q2 Calendar.quarter_of_year/3 + Formatter.abbreviated_quarter/1
QQQQ wide_quarter 2nd Quarter Calendar.quarter_of_year/3 + Formatter.wide_quarter/1
QQQQQ narrow_quarter 2 Calendar.quarter_of_year/3 + Formatter.narrow_quarter/1
YY two_digits_week_based_year 4, 14, 14, 14 Calendar.week_in_year/3
Y+ week_based_year 4, 14, 214, 2014 Calendar.week_in_year/3
w+ week_in_year 1, 9, 13, 42 Calendar.week_in_year/3
W+ week_in_month 1, 9, 13, 42 Calendar.week_in_month/3
E abbreviated_day_of_week Tue Calendar.day_of_week/3 + Formatter.abbreviated_day_of_week/1
EE wide_day_of_week Tuesday Calendar.day_of_week/3 + Formatter.wide_day_of_week/1
EEE narrow_day_of_week T Calendar.day_of_week/3 + Formatter.narrow_day_of_week/1
H+ hour (0-23) 1, 01, 23 struct.hour
h+ am_pm_hour (1-12) 1, 01, 11 struct.hour
a am_pm AM, PM struct.hour + Formatter.am_pm/1
m+ minute 1, 11, 59 struct.minute
s+ second 1, 11, 59 struct.second
S+ fraction_of_second 1, 001, 123456 struct.microsecond
VV time_zone Brasil/Sao Paulo struct.time_zone
zz zone_abbr BRT struct.zone_abbr
x zone_offset_basic_optional -08, +0530, +00 struct.std_offset + struct.utc_offset
xx zone_offset_basic -0800, +0530, +0000 struct.std_offset + struct.utc_offset
xxx zone_offset_basic -08:00, +05:30, +00:00 struct.std_offset + struct.utc_offset
X zone_offset_basic_optional_with_z -08, +0530, Z struct.std_offset + struct.utc_offset
XX zone_offset_basic_with_z -0800, +0530, Z struct.std_offset + struct.utc_offset
XXX zone_offset_extended_with_z -08:00, +05:30, Z struct.std_offset + struct.utc_offset

Whenever there is a plus sign at the end, it means the number of entries specifies the minimum number of digits. The exceptions are yy and YY, which is always shown as two digits regardless of how many digits, and S, which truncates.

Besides all entries above, we also support the following stand-alone formats: L for months (same as M) and q for quarters (same as Q).

The source column is used as a reference for the implementation and it won’t be present in the final documentation.

Rationale

Last but not least, it is worth discussing the rationale for date/time formatting. If you have an application that works with calendar types, it is likely that you have to format them at some point. If your application mostly interfaces with other systems, then there is a chance the built-in ISO format is enough, but not always. For example, some HTTP headers use a different format than the recommended ISO one. Therefore adding formatting to the standard library feels like a natural next step to the existing functionality.

Of course, there are some downsides to adding this functionality. First of all, there are many syntaxes for date/time formatting, and they all feel unnatural to some extent. While we could easily argue the ICU/Unicode is the one that makes the most sense for Elixir, as Elixir follows the standard in many other occasions, it is hard to argue it is the best approach generally (or even if there is such thing as best). One approach, which is orthogonal to the one above, is to allow the format to also be given as a list of atoms instead of cryptic single-letter definitions.

Another discussion, which may or may not impact this one, is about parsing. The parsing specification is often the same as the formatting specification but we have explicitly decided to not support parsing in Elixir. First of all, it is really hard to support a general but efficient runtime date/time parsing strategy. If you expect certain formats, it is almost always better to define functions that parse specifically those formats. Things get trickier if we consider the fact we need to support internalization, which is trivial for formatting, but quite expensive for parsing. In other words, while we can provide a general and efficient implementation for formatting, we can’t do so for parsing. Since different trade-offs can be made here, ranging from performance to flexibility, we are not comfortable in picking one or another.

Roadmap

We don’t plan to add this functionality directly to Elixir. Instead we will develop it as a library and collect feedback. The complexity of the implementation will also dictate if this will become part of core or not, but we believe the implementation will be relatively simple.

The first step is validation of this proposal and then a library will be developed as part of The Elixir programming language · GitHub for futher validation and feedback.

Feedback

Your turn.

Most Liked

kip

kip

ex_cldr Core Team

The full table of supported codes for ICU is in the reference you supplied, more specifically at https://www.unicode.org/reports/tr35/tr35-dates.html#Date_Field_Symbol_Table

E, EE and EEE are week day formats like “Tuesday, Tue, T”
MMM, MMMM are month names like “September, Sept”

There are others that are primarily textual and therefore locale specific like a, b, B that I recognise are outside of the scope of this proposal.

I’m just trying to work out if they remain valid formatting codes for this proposal, but not supported by the default formatter. Or if they are considered invalid codes for this proposal.

On the locales front - yes, your API strategy makes a lot more sense. The core of what i was trying to express, badly, is that even when a package or a language doesn’t support multiple locales, it implicitly is supporting one which for Elixir is en-US. Today, each package that intends to help support multiple locales like Gettext, Cldr, Trans has its own notion of current locale and what constitutes a supported locale, or locale name format. A unifying strategy for current locale and locale name would be helpful. But outside the scope of this proposal of course.

josevalim

josevalim

Creator of Elixir

Thanks everyone for the feedback!

I am bringing the proposal down for now because the discussion has made it clear that, if we want to support this, then we should go ahead and fully support ICU (with all formats, variants and what not) or we don’t support it all.

After all, if we only support part of it, then the community will have to implement everything again, like in @kip’s great projects, or we will have to define many interfaces so libraries are able to continue the work we started. None of those options feel great and then I would rather make sure that the Calendar functionality provides everything @kip needs in his libs.

So here are the next steps: we will look into simpler formatting syntaxes, possibly strftime, and see if we can write a proposal with reduced scope and less pitfalls that provides a minimum set of functionality without overlapping with more complex use cases. In case we can’t find anything that fits those constraints then we will give up on having formatting in core for now.

josevalim

josevalim

Creator of Elixir

Those are very good questions. We generally want Elixir core to be lean. We want the community to rely on third party packages. In fact, I have talked with @bitwalker about breaking Timex apart (for example, parsing/formatting could be its own lib).

The only possible rationale for including this in Elixir is because it belongs to the current feature set already part of the language and that this feature is small enough to warrant its inclusion in the language but I agree it is a blurry line.

To avoid any debate, I will just follow whatever Unicode and/or CLDR recommends as default. You will be able to customize by passing a third argument to format.

I haven’t thought about this yet and to be honest, I don’t think so. A timezone database is meant to provide the same results. What would change between them is functionality such as live updates, performance, etc. So while it is global, it is not mean to cause a difference in behaviour.

Setting a different formatter will change behaviour. So for now I would keep it explicitly opt-in.

Qqwy

Qqwy

TypeCheck Core Team

Interesting!
I can see nothing wrong with the proposal itself; it seems very well-written, clear and well-defined in scope. However, I do have one question to ask as “devil’s advocate”:

There exist widely-used libraries such as Timex that have a very complete implementation of formatting (as well as parsing). (And I presume the proposal is based on this).

So why is this something Elixir (specifically speaking, the ‘elixir-lang’ team) should concern itself with? What would the advantage of an official library be in this case, and would it be strong enough to warrant the creation and maintenance of this library?

:slight_smile:

Where Next?

Popular in News Top

Elixir
Release: https://github.com/elixir-lang/elixir/releases/tag/v1.9.2 1. Enhancements Mix [mix release] Allow {:from_app, app_name} as a v...
New
Elixir
Release: https://github.com/elixir-lang/elixir/releases/tag/v1.11.2 1. Bug fixes Elixir [Code] Do not crash when getting docs for missi...
New
josevalim
Announcement: https://elixir-lang.org/blog/2019/01/14/elixir-v1-8-0-released/ Release notes: https://github.com/elixir-lang/elixir/relea...
New
Elixir
1. Enhancements Elixir [File] Support distributed File.Stream [Module] Add Module.get_last_attribute/3 [Task] Reduce footprint of tasks ...
New
Elixir
Official announcement: Elixir v1.15 released - The Elixir programming language This release requires Erlang/OTP 24 and later. Elixir v1...
New
whatyouhide
Hello folks, I am super excited to finally share what I've been working on in the last few months: StreamData, an Elixir library for da...
New
josevalim
This is a small release with important bug fixes for those using Cover and Dialyzer on Erlang 19. See the release notes for more info: ...
New
josevalim
See the release notes: https://github.com/elixir-ecto/ecto/releases/tag/2.2.0-rc.0 To try it in your projects: {:ecto, "~> 2.2.0-rc"...
New
Elixir
Release: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0-rc.0 Support for Erlang/OTP 21+ Elixir v1.10 requires Erlang/OTP 21+...
New
josevalim
Hello everyone, I would like to propose the addition of the Registry project to Elixir: https://github.com/elixir-lang/registry The...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement