I might also just start a plain IEx console and leave that running and see if the time drifts as that would point to it being M1/OS related. I wonder if when the laptop sleeps that could be causing the issue, that’s almost certainly the case each time I notice this. Of course I wasn’t taking detailed notes, but I can’t think of a time where i’ve started the server, and not had some reason that the laptop was sleeping for a few hours, and its usually after that that I notice it, but again this is just from vague recollection.

Agreed. The underlying issue is almost certainly something in OTP rather than Guardian, but I still think there’s value in switching from system_time to os_time in Guardian, no?

Also, interestingly, my IEx session has drifted back to 0.

iex(1)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
0
iex(2)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
196
iex(3)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
196
iex(4)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
197
iex(5)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
196
iex(6)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
197
iex(7)> 197 / 60
3.283333333333333
iex(8)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
195
iex(9)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
17
iex(10)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
17
iex(11)> DateTime.to_unix(DateTime.utc_now()) - System.system_time(:second)
0

Yeah it’s a thing in Erlang because most runtimes are relatively ignorant and are not trying to provide quite the same guarantees around scheduling.

I think in this case I think the original time abstraction was simply insufficient and maintaining that abstraction in the face of precision time and wall clock warping led to some slow syncing / unwanted consequences.

But things are good now with the new time API AFAIKT.

I understand that there is some deeper cause. You may be right about the NIF claim. I do have some NIFs, and am running some ML models via Nx/EXLA, which uses a lot of NIFs, but I guess my other question still remains, if we have the following two options:

1. Timestamp 1, usually reliable, but sometimes for certain reasons can drift from the real time, maybe sometimes by a few seconds, but also sometimes by 97 minutes

2. Timestamp 2, reliable, doesn’t seem to drift from the real time

What would be the reason to ever use timestamp 1?

I’ve only seen it on my M1, but as it’s still in development, it hasn’t had any real usage/runtime outside of the M1. I do have a server it’ll be running on, though it’s not running right now. I’ll start that up, and maybe start a task that trickles calls that would hit this nif and the ML models for 24 hours and see what the server reports. I’ve had this issue happen fairly consistently though locally. It seems to be related to time the server is running, rather than usage, e.g. if I start the server and then ~12 hours later try to make a request from the client, it’ll have the “expired JWT” issue(not because the JWT is 12 hours old, but rather because the newly created access token is expired), until I restart the server, but I won’t get that even after say 4-6 hours of development where the NIF would be getting called more often.

It is also a fairly different from typical setup, because the project relies on a few ML models, it’s running Bumblebee which starts up a couple models via some NIFs and EXLA, erlport which is running some python code, and the Rust NIF from above. Which certainly isn’t the most common use case, but still if a change from system_time to os_time can fix the issue for even a few people, it seems useful. I’ll start a issue on Github to see if there’s a reason for the use of one over the other.