Hi,

I work at InfluxData, the company behind InfluxDB.

We’re actively looking at how we can contribute to the BEAM ecosystem, so it’s great to see some people working with this stack use InfuxDB.

I generally advise avoiding pre-aggregated metrics, but obviously there’s a trade off and sometimes it’s the best route forward.

You can do this currently with Telegraf, which can proxy InfuxDB; this would allow you to do aggregation without code.

I’ll put an example together if you think it would be useful

Is InfluxData involved at all with https://opentelemetry.io/? The first draft of the metrics specification is being finished up now I believe.

If you are involved in OpenTelemetry or plan to support it then I think the best way for InfluxData to contribute to the BEAM ecosystem is through the Erlang/Elixir libraries, https://github.com/open-telemetry/opentelemetry-erlang

We have a SIG as part of the OpenTelemetry project https://github.com/open-telemetry/community#erlangelixir-sdk

I guess the solution for doing it within the telemetry ecosystem would be have the telemetry reporter do aggregation/sampling. Afaik it’s the place meant to handle conversion between reported metrics and what’s actually send out to elsewhere. This can be as easy as just forwarding data or be quite complex by doing pre-aggregation or sampling. In the end it depends on how many moving parts one likes to have inside and/or outside of telemetry.

Yes, a proper place would be a reporter if one wants to integrate with Telemetry.Metrics, which provides nice abstraction over how events should be aggregated.

The InfluxDB reporter mentioned above takes a different approach where all events are pushed directly to InfluxDB, something that @rawkode suggested. The benefit of doing that instead of pre-aggregating is that we don’t need to know in advance what aggregations we’re going to run in order to analyze the data. As always, there are tradeoffs: pushing every event might be very bandwidth consuming, but aggregating in-process may consume considerable amount of memory.

This is the difference between logs and metrics, more can be read in this article by Grafana