Excellent question. I think that a proper knowledge base on OTel in Elixir is long overdue for whoever might have the time to author it.

Just to make it easier to get into using any telemetry backend, one might take a look at telemetria library which is backend-agnostic and allows telemetry adoption via @telemetria module attributes, like

defmodule Forecast do
  use Telemetria

  @telemetria level: :info, group: :weather_reports, locals: [:farenheit]
  def weather(city) do
    fahrenheit = ExternalService.retrieve(city)
    Converter.fahrenheit_to_celcius(fahrenheit)
  end
end

I just set up a telemetry stack for a Phoenix Application. Its on my homelab, so I’m not sure how it’ll work on fly.io, but hopefully it’ll give you a place to start. For traces you will need, at minimum, opentelemetry, opentelemetry_api, and opentelemetry_exporter. On top of that there are packages like opentelemetry_phoenix, which will trace important libraries for you out of the box. For metrics Elixir’s OTEL SDK isn’t ready yet, but you can use prom_ex, which is really good. And you can use an OTEL collector to handle both those metrics and those traces, and then forward them to whatever tool you are using. For logs you’ll just have to use whatever tool to collect them your stack offers (like promtail for grafana loki). Hopefully that’s enough to start, but if you have any questions just ask.

• GitHub - openobserve/openobserve: 🚀 10x easier, 🚀 140x lower storage cost, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces, RUM, Error tracking, Session replay).
• GitHub - SigNoz/signoz: SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool

I have used the first successfully, at least locally. Should be fine for production usage but do your own extra research on that.

Finally wrote teh article over here: How to Use OpenTelemetry, PromEx, and Promtail to Build an Observability System with OpenObserver Exporter in Elixir