If the data doesn’t have interdependencies then I’d suggest using multiple modules (each in it’s own file). This way you can leverage the compiler of elixir, which allows you to compile modules in parallel. You could still have one central module, which delegates to the actual implementations in the modules with the data compiled into.

Another step I’d consider here is actually compiling data not as a blob into the modules, but if possible compile it into multiple functions with different function heads, which could lessen the runtime load on iterating big chunks of data over and over again. You can look at ex_cldr for inspiration, which does exactly that with the cldr database.

E.g. for your zips compile into function like this:

@external_resource Path.join(__DIR__, "../../../priv/zip_cities.etf")
zip_cities = File.read!(Path.join(__DIR__, "../../../priv/zip_cities.etf")) |> :erlang.binary_to_term()
for {zip, city} <- zip_cities do
  def zip_city(unquote(zip)), do: unquote(city)
end

As far as I remember, mix escript.build requires to strip the modules as well as compressing them every time into the binary. Depending on the physical size of the modulefiles this can of course take a long time.

As a rule of thumb, building the escript will always take longer than tarring the _build/$env/lib/*/ebin folders.

Thanks for the advice… very good idea to put each data file in a different module to leverage the parallel compiler!

I’ve seen threads on here talking about generating thousands of functions in the same module which I thought were interesting but I’ve never benchmarked that. If I get some time I can compare that to my Map.get based approach. One of my maps has ~44,000 keys (zip codes in US) and another has ~180,000 keys (unique city/st in US) and the Map.get performance is still quite good.

I solved my initial problem by extracting the ZipCityData into a separate umbrella app/library and including it as a dependency. It rarely changes so my IEx recompiles/tests when not changing that module are super fast again. Of course mix escript.build is slow but that’s OK as I don’t release new versions very often.