I am running some design tests. The example here is not a real system but just some rough parts to test how things can fit together.

APPLICATION STRUCTURE

Hypothetically imagine a structure like this:

     APPLICATION
     |__SUPERVISOR
        |__SUPERVISORTEST
           |__DATACACHETEST
        |__WEBSOCKET

The DataCacheTest is a GenServer that starts an ETS table which can be read/written by anyone anywhere. Read/write is not passed through the call/cast system before the ETS. Yes, there are race conditions.

WebSocket is an HTTP router and/or websocket upgrade by which users might access or manipulate the data in the ETS table.

PROBLEM

The problem I am trying to figure out is:

What happens if the DataCacheTest GenServer crashes? It will be restarted, but in the mean time, queries on the ETS data from the HTTP system will fail. (ETS table won’t exist.) What happens in that moment? ie. While the ETS table GenServer is restarting?

1) CHECK IF EXISTS?

One idea is you could do a check on the GenServer existing like inside DataCacheTest:

def exists do
        pid = Process.whereis(__MODULE__)
        is_pid(pid) # return true or false
end

And then use that before any read/write operations on the module. ie. running in websocket:

if DataCacheTest.exists() do
    DataCacheTest.read_value()
end

But I feel like this is wrong. Because you could theoretically pass the exists() check and then still fail at the read_value() if it dies between that time. Or it dies but the Registry isn’t updated yet.

You don’t want the error reaching the HTTP socket module as I’m not sure how crashing works but you certainly don’t want this to crash as all your users would be disconnected.

2) TRY/RESCUE ALL ETS OPERATIONS?

My best idea is then to (knowing that any ets operations can fail in this case) wrap any :ets.lookup and :ets.insert functions inside DataCacheTest like so:

    result = try do 
        :ets.lookup(:doesntexist, :doesntexist) 
    rescue 
         ArgumentError -> IO.puts("Caught ETS failure") 
        @failure_value
    end

ie.

    try do :ets.lookup(:doesntexist, :doesntexist) rescue ArgumentError -> IO.puts("Caught ETS failure") end
    try do :ets.insert(:doesntexist, {:doesntexist, :whatever}) rescue ArgumentError -> IO.puts("Caught ETS failure") end

These two operations do seem to catch the errors without the red error test when I test them in the console. I’m guessing any time you see red error text, something may be crashed? I’m not sure. Is that the case?

Is this the right idea?

Thanks for any thoughts.

TEST CODE

1) Application

Hypothetical Application that starts the system hierarchy above.

defmodule MyApplication.Application do

    #use Application
    def start(_type, _args) do
        web_socket = {Plug.Cowboy, plug: TestWebSocket, scheme: :https, options: [ port: 4100, ] }
        supervisor_test = {SupervisorTest, nil}
        children = [web_socket, supervisor_test]
        opts = [strategy: :one_for_one, name: MyApplication.Supervisor]
        supervisor = Supervisor.start_link(children, opts)
    end
end

2) ETS Supervisor

Not really needed, I suppose, but I was just thinking maybe it is nice to have a different supervisor here for the ETS Genserver so tested creating this.

# test run as Supervisor.start_link([{SupervisorTest, nil}], strategy: :one_for_one)
defmodule SupervisorTest do
    use Supervisor

    def start_link(args) do
        Supervisor.start_link(__MODULE__, args, name: __MODULE__)
    end

    # how to initialize supervisor on start up
    def init(_args) do
        children = [
            {DataCacheTest, nil}
        ]
        Supervisor.init(children, strategy: :one_for_one)
    end

end

3) ETS Table Owner

This GenServer creates an ETS server and has some static (non-call/cast) functions for accessing that data anyone anywhere can use.

defmodule DataCacheTest do 

    # Supervisor.start_link([{ DataCacheTest, nil}], strategy: :one_for_one)
    # DataCacheTest.read_value()
    # DataCacheTest.increment_value()
    # DataCacheTest.crash()

    use GenServer

    @key_name :ets_key
    @value_name :ets_value
    @table_name :data_cache_test_ets_table
    @default_value 1

    def start_link(args) do
        GenServer.start_link(__MODULE__, args, name: __MODULE__)
    end

    #===============================================
    # (i) INITIALIZATION FUNCTION (CREATE ETS TABLE)
    #===============================================
    def init(args) do
        create_table()
        {:ok, nil}
    end
    def create_table() do
        case :ets.info(@table_name) do
            :undefined ->
                :ets.new(@table_name, [:set, :public, :named_table, {:read_concurrency, false}, {:write_concurrency, false} ])
                # RETURNS :global_ets_table
            _->
                nil
        end
    end

    #=====================================
    # ii) ETS ACCESSOR FUNCTIONS - intentionally not using call/cast and not atomic, yes there will be racing
    #=====================================
    def read_value() do
        result = :ets.lookup(@table_name, @key_name) #=== WRAP IN TRY/RESCUE???
        case result do
            []-> #doesn't exist, insert default
                :ets.insert(@table_name, {@key_name, @default_value})  #=== WRAP IN TRY/RESCUE???
                @default_value
            [{@key_name,  value}]-> #found key/value
                value
            _->
                #not sure why we are here, create table again maybe
                IO.puts("FAILED READING VALUE")
                @default_value
        end
    end
    def increment_value() do
        current_val = read_value()
        new_val = current_val + 1
        :ets.insert(@table_name, {@key_name, new_val})  #=== WRAP IN TRY/RESCUE???
        new_val
    end

    #================
    # (iii) CRASH
    #================
    def crash() do
        pid = Process.whereis(__MODULE__)
        Process.exit(pid, :crash)
        raise "intentionally crashed"
    end
end

4) Websocket/Server

This wants to run queries on the ETS Data directly without going through call/cast

defmodule TestWebSocket do
    use Plug.Router

    plug Plug.Logger
    plug :match
    plug :dispatch

    get "/get_value" do
        value = DataCacheTest.read_value() # WHAT HAPPENS IF ETS GENSERVER IS CRASHING?
        conn
        |> put_resp_content_type("text/plain")
        |> send_resp(200, to_string(value))
    end

    get "/increment" do
        value = DataCacheTest.increment_value()  # WHAT HAPPENS IF ETS GENSERVER IS CRASHING?
        conn
        |> put_resp_content_type("text/plain")
        |> send_resp(200, to_string(value))
    end

    match _ do
        send_resp(conn, 404, "not found")
    end
end