the_wildgoose

the_wildgoose

Reading mtime (stat) to microsecond accuracy

I tried using

  • File.state(“some/file”)

and discovered that it truncates the timestamps to the nearest second. It seems to be a wrapper for the underlying erlang call:

  • :file.read_file_info(‘some/file’, [{:time, :posix}])

…which returns times in whole seconds…

Any suggestions on how to most performantly access mtimes to maximum accuracy of the filesystem?

Equivalently, the use is debouncing reads of a small JSON file, so a possible workaround is a high speed way to checksum the file? Suggestions?

(The file is small, we have a problem (insert reasons) that means we need to acquire an expensive lock (100s of milliseconds) before reading it, so I need robust way to detect if it’s been changed since last read. Yes, long term we can work on improving the speed of locking, but…)

Most Liked

BradS2S

BradS2S

Here’s a NIF:

#include <erl_nif.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>

//  detect operating systems that I know something about
#ifdef __APPLE__
    #define PLATFORM_MACOS
#elif __linux__
    #define PLATFORM_LINUX
#endif

// Useful resource:
// https://andrealeopard.com/posts/using-c-from-elixir-with-nifs/#defining-a-nif


// ERL_NIF_TERM is a "wrapper" type that represents all Erlang types
// (like binary, list, tuple, and so on) in C.
static ERL_NIF_TERM get_file_time(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
{
    char file_path[1024];
    long seconds, nanoseconds;

    
    if (enif_get_string(env, argv[0], file_path, sizeof(file_path), ERL_NIF_LATIN1) <= 0) {
        // erl_nif.h provides several enif_make_* functions
        // to convert C values back to Erlang values.
        return enif_make_badarg(env);
    }

    struct stat filestat;
    if (stat(file_path, &filestat) == 0)
    {
        #ifdef PLATFORM_MACOS
        seconds = filestat.st_mtimespec.tv_sec;
        nanoseconds = filestat.st_mtimespec.tv_nsec;
        #elif defined(PLATFORM_LINUX)
        seconds = filestat.st_mtim.tv_sec;
        nanoseconds = filestat.st_mtim.tv_nsec;
        #else
        seconds = filestat.st_mtime;
        #endif
        
        return enif_make_tuple2(env,
                                // A enif_make_atom
                                enif_make_atom(env, "ok"),
                                enif_make_tuple2(env,
                                                 // Here's a enif_make_long
                                                 enif_make_long(env, seconds),
                                                 enif_make_long(env, nanoseconds)
                                                 )
                                );
    }
    else
    {
        // Here's a enif_make_tuple{n}
        return enif_make_tuple2(env,
                                
                                // https://stackoverflow.com/questions/503878/how-to-know-what-the-errno-means
                                enif_make_atom(env, "error"),
                                enif_make_int(env, errno)
                                );
    }
}

// Let's define the array of ErlNifFunc beforehand:
static ErlNifFunc nif_funcs[] =
{
    // {erl_function_name, erl_function_arity, c_function}
    {"get_file_time", 1, get_file_time}
};

// We now have to export the function we wrote to Erlang.
// We'll have to use the ERL_NIF_INIT macro. It looks like this:
ERL_NIF_INIT(Elixir.FileTime, nif_funcs, NULL, NULL, NULL, NULL);

I compiled it using this command:

gcc -fPIC -I/usr/local/lib/erlang/usr/include/ \
     -dynamiclib -undefined dynamic_lookup \
     -o file_time.so file_time.c

Here’s the module I wrapped it in:

defmodule FileTime do
  @on_load :load_nifs

  def load_nifs do
    :erlang.load_nif("./file_time", 0)
  end

  def get_file_time(_file) do
    raise "NIF get_file_time/1 had an error. :("
  end
end

(If you do a string it won’t work.)
{:ok, {seconds, nanoseconds}} = FileTime.get_file_time('file_as_char_list.txt')

{:ok, datetime} = DateTime.from_unix(seconds)

I tested it on a couple files and seemed to be working ok.

jhogberg

jhogberg

Erlang Core Team

Relying on time stamps, however accurate, is not a reliable way to check whether files have changed. Neither are content comparisons (checksums or otherwise) if there’s a possibility of the file being modified while you’re reading it. Try writing a NIF that uses inotify(7) or similar instead and listen for changes.

I have a feeling that you’re trying to solve the wrong problem though, why are you communicating over the file system?

the_wildgoose

the_wildgoose

I’m using inotify. I need to debounce multiple notifications from it.

Can I just add that I’m not hoping for an argument on the merits of mtimes and comfortable with their limitations.

Larger situation is that multiple processes across an embedded system are all editing a master JSON config file. Think of something like openwrt and it’s config system. You need 2 basic primitives, open with a shared lock and open with an exclusive lock. Due to the varied different implementations/programming languages that I need to support, I’m using flock to implement my locking. For various reasons the required incantations to support flock using exec under elixir are “slow”. This is due to calling the native “flock” binary, which itself needs to bring up a shell to be safe against process failure.

Longer term I will rewrite flock in rustler. I already wrote an implementation in zigler, but that seems to have lost traction and whilst the code runs fine on a number of platforms, it fails to execute correctly on arm32. Development time is the limitation here. I also see that there is a pull request for OTP to implement flock in OTP27, which should land in about 14 months or so. So I just need to get this project stood up until then

So, back to the original problem: I have a database file, its slow to lock it for (synchronised) reading. I can read it’s contents raw without a problem, eg to do a CRC check on it, or I can check it’s mtime to see if it changed since I last read it. The file is read regularly, but written very rarely. Goals are to avoid calling flock on every read. Cache is blown away on every inotify and when elixir alters the file. However, I’m still getting stale reads in some corner case. I’m looking to implement a safety net that also re-reads if the file has changed based on mtime (or perhaps something else)

OK, is the problem space clear enough? Any suggestions on accessing mtime that are neater than running exec “stat” and parsing the output?

D4no0

D4no0

Since nowadays hashing is so common, there is a chance that the CPU you are using might have acceleration for some specific hashing algorithms. If I were to choose, I would go CRC at the beginning.

dimitarvp

dimitarvp

I’m interested what’s your actual scenario that you eventually ended up with having to modify a JSON file and checking if you modified it recently.

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New

We're in Beta

About us Mission Statement