garrison
Is anyone working on "AI Agents" in Elixir?
For those who are not aware, “AI agents” are, for the most part, commodity LLMs which are given access to “tools” and prompted to complete tasks, possibly in some sort of loop.
The tool use is facilitated by a program which scans the output text of the LLM and looks for a “tool call” request (in some standard format), and then executes that call. For example, you might give the model access to a “calculator” tool which enables it to do math, or a “weather API” tool to check the weather. And so on. The model is given a prompt which tells it what tools it has access to, and I believe most models coming out nowadays are trained to some degree on tool use so that they get the general idea.
The “agentic” behavior here is somewhat arbitrary, but the idea is that you have some sort of feedback loop. The model generates a tool call, receives the result, and then perhaps generates more calls based on that result. People have been using this to write code, for example, with (so far) limited success.
The current emerging “killer app” for agents is the “deep research” model, which has been adopted by google, openai, perplexity, twitter (lol), and so on. The basic idea here is that you give the model a “search engine” tool and then just prompt it to run in a loop searching, reading results, and then coming up with more searches. Then it generates a nice summary (“report”) at the end for human consumption. It goes without saying that this task is a lot easier than writing code, and as a result agents seem to be actually “catching on” for the first time.
Due to the autoregressive nature of current LLMs, which has proved to be quite sticky thus far, they perform extremely poorly for “local” use. Current autoregressive models require the entire model to be run through the GPU’s registers on every forward pass just to generate one token. As a result, “local” inference is completely bottlenecked by memory bandwidth. If you have a 30GB model (on the low end of “useful”), and a GPU with 600GB/s memory bandwidth (that’s pretty good), you would expect 20 tokens/sec (fairly usable). Unfortunately GPU memory bandwidth is expensive and 30GB is not enough for a top tier model.
However, this problem vanishes with batching. GPUs are built for parallel compute, and deep nets are built to utilize it. If you batch, say, 10 requests at a time, all of a sudden you are getting 200 tokens/sec on the same hardware (flops notwithstanding). The point being: there is a forcing function towards multitenancy. This is why everyone is using cloud APIs instead of running their own models - the cost reduction is enormous.
What this means is that “AI agents” are actually just glue code for interacting between LLM APIs and “tool” APIs. And that’s where Elixir comes in: we are very good at soft-realtime. Elixir and the BEAM are the ideal ecosystem for this. LiveView is the perfect tool for server-side realtime UI. If you were going to build some sort of “agentic” app, this would be the platform.
So I’m curious, is anyone doing something in that space?
Most Liked
mikehostetler
Author of Jido here
I’m actively wrestling with these ideas. I’ve implemented several “Applications” with Jido now that … after finishing them … I really struggle to answer whether they are better with Jido or not. Long term - a few GenServers that wrap req calls to LLM API’s are better.
I have this overwhelming feeling that Jido is the right direction - but has not arrived at a sensible destination.
A few other thoughts to share:
- There’s a complexity vector here - a simple LLM wrapper doesn’t need a sophisticated agent framework - Oban works great.
- Most agent implementations are really really simplistic - any dreams of massive swarms of agents demonstrating collective intelligence are still dreams - Elixir is more suited to larger swarms of agents due to OTP
- While implementing Jido, I learned a lot about OTP - the educational journey was amazing. I found Joe Armstrong’s blog while on this journey - and realized that some of the features of Jido are simply constrained implementations of OTP - I don’t think that’s bad necessarily - but probably not the best implementation
- While it’s easy to write an run 10,000 agents with Jido - it’s not that useful - for all of the normal distributed systems problems that come from running and coordinating 10,000 GenServers
There’s more questions then answers right now - but I do think LLM’s are here to stay so it’s better to wrestle with them
This particular space in our industry is evolving a lot right now - so I’m content to just continue wrestling and playing with the ideas. A few “first principles” I’ve collected so far:
- Agents will be a new buzzword for “LLM Applications”
- Agentic workflows are just workflows - low volume ETL pipelines with more variety
- Multi-agent is simply a new variant on distributed systems problems
- Elixir is well-suited for this - but suffers from a slower iteration cycle - so I’ve been following Python and TS “agent” frameworks closely and pulling in patterns to Jido that I feel will be durable over time
joelpaulkoch
nallwhy
Really enjoying this thread—so many insightful takes. I’d love to add a perspective from someone currently building a service on top of Ash and Ash AI(GitHub - ash-project/ash_ai: Structured outputs, vectorization and tool calling for your Ash application), where we’re integrating an agent-style chatbot into a real workflow.
One thing I believe strongly:
An agent’s job is to elevate and clarify user intent, then communicate and act on it to drastically simplify the UX.
In the app I’m building, the flow looks something like this:
a user uploads a contract file and just says “process this”.
The agent then:
- reads the file and extracts key data
- checks if the referenced client is already in the system
- creates the client if necessary
- continues to create an invoice→ all while mixing automation with UI-driven confirmation, so the user stays in control.
Internally, this feels incredibly natural and surprisingly fluid .
Where Elixir shines in this setup:
- Ash Framework: The declarative power of Ash is hard to overstate. It lets me define tools cleanly and declaratively, which plug seamlessly into AshAI. This means I can expose my application logic as “tools” with almost no extra effort.
- LiveView or Channels: Real-time interaction with the user is crucial. For example, once the agent identifies what kind of data needs to be confirmed, we bring the user to a LiveView-powered form to review/edit inputs. Once confirmed, the agent resumes and continues the workflow. This kind of multi-step, multi-view interactivity is where LiveView or channels make things incredibly smooth.
I’m still early in the journey, but this combination of structured domain logic and real-time agent orchestration feels like a powerful direction.
garrison
Protocols aside, the big advantage for Elixir here is that our ecosystem was practically made for this. There is no better platform for writing soft-realtime glue code between different models/services/APIs.
If you take a step back and think about this it makes perfect sense: Erlang/OTP were literally designed for telecommunications. This problem, facilitating communication between models/services/users, is telcom, it’s just that the scope has expanded far beyond phone calls. It is a testament to the wisdom and creativity of those who built these systems that they can still be so relevant today.
“Agentic” apps built with Elixir will scale better with much lower latency (especially tail latency) than anything built with Python, and (IMO) developer ergonomics are much better too, though Python is far from the worst.
If anything, our biggest “competitor” will probably be JS simply because a couple of large companies (e.g. Cloudflare) have committed to in-process multitenancy which will drive costs down significantly for those who aren’t serving enough to saturate a VM core. On the other hand, one might argue those customers aren’t very valuable.
mikehostetler
I love talking and thinking about this stuff! The depth of conversations I’ve had from other community members makes me appreciate Elixir even more
I think it’s more important that we provide “building blocks” in Elixir rather than an “agentic framework” per se, since the BEAM itself provides that framework.
Yes, I’m 100% with you here - this is why I was opinionated about keeping any specific LLMs out of Jido core. LLM’s are implemented as specific Agents, Skills and Actions in the jido_ai package - I was pretty happy with how well this worked
Yeah, I don’t see the value proposition for “swarms of agents” or anything like that
Short term - no - I don’t know of any examples deploying many unique agents right now. That said, I see a natural evolution where a product implements a 1-to-many agent pattern where one agent orchestrates the work of other agents on behalf of a user. This was the breakthrough that Manus demonstrated so well.
I’m not convinced this is good though - as you said it gets very anthropomorphic and too magical. Interestingly, I have asked my kids about this (ages 15 & 17) and they have less of an issue - so I’m acknowledging some personal bias here.
I think it’s best not to get distracted here.
Solid point - I pushed out the Jido weather example and went on holiday for Spring Break - which has prompted a lot of this reflection. I’ve been soaking up other example frameworks, reading everything I can get my hands on etc
This was a great read as well.
It’s been easy to get lost in the weeds - I’m back to building agents this week with some fresh perspectives and will be shipping more soon!
Popular in Dev Env & Tools
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"