ETS keeps its data entirely separate from all processes so when you access an ETS you copy the between the table and the process. HOWEVER you do not copy the whole table each time only the elements you actually access. This does mean you should not store all the data in one element but in multiple elements and you the key to select which element.

Actually there is no way to avoid copying data if you are sharing data between processes, keeping it in an ETS table or storing it somewhere “outside” the erlang/elixir system. It’s a fact of life. Just make sure you can access it in small chunks.

Depending on how often your shared data does change maybe look at this one: https://github.com/discordapp/fastglobal

It becomes less of a surprise if you think through the implications of doing that. Erlang does garbage collection on the process level, which is very simple in multiple regards: process memory is small, and processes are interruptable so a small pause to do a quick GC is acceptable. This keeps GC simple. Now, think of the case when you would get data by reference out of a process (an Agent is just a process) - suddenly the GC has to keep track of pointers globally in the VM and simplicity gets tossed out of the window.

(the actual details are, of course, a bit more complicated than I just said. This seems to be a decent quick overview with pointers to further reading. Erlang will ask you to open the hood and look at how the engine works a bit sooner than other systems, but the pay-off is good performing stable code and the investment isn’t that high (compared to, say, learning the Java Memory Model). Well worth it.)

You may have your reasons for not divulging any details about your processing requirements - so it is possible that you are absolutely correct that you need a separate copy of the data for each user - and then again that may simply be a superficially convenient choice.

How many simultaneous requests do you reasonably expect to be running against the data set? How large is the result that the requestor is going to get back?

In the BEAM environment it may make more sense to separate the “request” logic from the user’s client code and instead use that logic to build a short lived process that runs the logic against it’s own copy to produce the result. While that may lead to more copying it may actually require far fewer simultaneous copies of the data in memory and process termination makes GC extremely simple.

To cut down on the copying, processes could be reused a finite number of times or indefinitely (i.e. process pools as already suggested).

Copying could also be reduced/eliminated by partitioning the data in some logical way so that it could be efficiently accessed and shared (as already mentioned via ETS and/or distributing data dependent processing among multiple processes).

Shouldn’t be a surprise:

Erlang Programming 2e: Introduction: p. xiii:

Erlang belongs to the family of functional programming languages. Functional programming forbids code with side effects. Side effects and concurrency don’t mix. In Erlang it’s OK to mutate state within an individual process but not for one process to tinker with the state of another process. Erlang has no mutexes, no synchronized methods, and none of the paraphernalia of shared memory programming.

Processes interact by one method, and one method only, by exchanging messages. Processes share no data with other processes. This is the reason why we can easily distribute Erlang programs over multicores or networks.
When we write an Erlang program, we do not implement it as a single process that does everything; we implement it as large numbers of small processes that do simple things and communicate with each other.

The same essentially applies to Elixir. Sharing is convenient but that convenience comes at a cost - it’s all about tradeoffs. Furthermore some find the utility of Agents questionable while most see them as limited, see this recent topic.

Agent is a specialization that focuses entirely on state. GenServer embodies the more general notion of a process minding it’s own state and maintaining full control over access (via messaging) and mutation of that state.
In Elixir the Task is often used for short-lived processes but GenServers will still be used for one-off processing when multiple processes have to coordinate processing in the pursuit of a common objective.

GenServer is the fundamental building block, Task and Agent are mere convenience specializations that are typically only useful under the most trivial of circumstances.

Compilation means transforming stuff from user-friendly to machine-executable format. Read the Yaml, convert it to Erlang/Elixir/LFE code, then compile that source, and you have executable code - which seems to be what you want.

Usually, when you find you’re writing an interpreter, you’re doing it wrong. See also: Ruby