News>Announcing

Explorer - Series (1D) and dataframes (2D) for fast and elegant data exploration in Elixir

#explorer
billylanchantin

billylanchantin

Explorer - Series (1D) and dataframes (2D) for fast and elegant data exploration in Elixir

The Explorer library has been out for some time. But we just released the latest version (see below), and we thought we’d start posting updates to the forum.

Wait, Explorer is new to me. What are Series and DataFrames?

Explorer is a DataFrame library for Elixir.

DataFrame libraries are common in languages which have a focus on data manipulation, including:

If you’d like a more in-depth tutorial, there’s an excellent LiveBook called Ten Minutes to Explorer that you can play with:

But we’ll provide a quick overview here.

Briefly, you can think of a DataFrame like an in-memory table. Its purpose is to facilitate common data exploration and analysis tasks. As such, it’s a column-oriented table.

Column-oriented tables

If you’re unfamiliar with column-oriented tables, suppose you have a table of pet data like this:

type age color
cat 5 black
dog 2 brown
dog 3 brindle

A row-oriented organization of that data might look like this in Elixir:

rows = [
  [type: "cat", age: 5, color: "black"],
  [type: "dog", age: 2, color: "brown"],
  [type: "dog", age: 3, color: "brindle"],
]

It matches the original table fairly one-to-one. But the column-oriented version might instead look like:

columns = [
  type: ["cat", "dog", "dog"],
  age: [5, 2, 3],
  color: ["black", "brown", "brindle"]
]

It has same information, but “transposed”.

Column-orientation is beneficial if you’re asking questions that require a lot of number-crunching like “What’s the average age of all pets?”. In the row-oriented version, finding the average age would require first looking through the entire contents of the table to collect the relevant data. But in the column-oriented version, those values have already been co-located in memory.

Series and DataFrames: columns and tables

In dataframe parlance, a “series” is a single column and a “dataframe” is a collection of named series, aka a table.

Our example above would look like this:

type = Explorer.Series.from_list(["cat", "dog", "dog"])
age = Explorer.Series.from_list([5, 2, 3])
color = Explorer.Series.from_list(["black", "brown", "brindle"])

df = Explorer.DataFrame.new(type: type, age: age, color: color)
# #Explorer.DataFrame<
#   Polars[3 x 3]
#   type string ["cat", "dog", "dog"]
#   age s64 [5, 2, 3]
#   color string ["black", "brown", "brindle"]
# >

Some things to note:

  • Each series has a corresponding data type or “dtype”, e.g. type has the dtype string.
  • The word “Polars” appears. That indicates that this dataframe is using the backend powered by the fantastic Polars library (the default backend).

And if we really did want to know the average age of the pets, that would look like this:

Explorer.Series.mean(df["age"])
# 3.3333333333333335

Features and design

Preiminaries out of the way, here are Explorer’s high-level features:

  • Simply typed series: :binary, :boolean, :category, :date, :datetime, :duration, floats of 32 and 64 bits ({:f, size}), integers of 8, 16, 32 and 64 bits ({:s, size}, {:u, size}), :null, :string, :time, :list, and :struct.

  • A powerful but constrained and opinionated API, so you spend less time looking for the right function and more time doing data manipulation.

  • Support for CSV, Parquet, NDJSON, and Arrow IPC formats

  • Integration with external databases via ADBC and direct connection to file storages such as S3

  • Pluggable backends, providing a uniform API whether you’re working in-memory or (forthcoming) on remote databases or even Spark dataframes.

  • The first (and default) backend is based on NIF bindings to the blazing-fast polars library.

The API is heavily influenced by Tidy Data and borrows much of its design from dplyr.
The philosophy is heavily influenced by this passage from dplyr’s documentation:

  • By constraining your options, it helps you think about your data manipulation challenges.

  • It provides simple “verbs”, functions that correspond to the most common data manipulation tasks, to help you translate your thoughts into code.

  • It uses efficient backends, so you spend less time waiting for the computer.

The aim here isn’t to have the fastest dataframe library around (though it certainly helps that we’re building on Polars, one of the fastest).
Instead, we’re aiming to bridge the best of many worlds:

  • the elegance of dplyr
  • the speed of polars
  • the joy of Elixir

That means you can expect the guiding principles to be ‘Elixir-ish’. For example, you won’t see the underlying data mutated, even if that’s the most efficient implementation. Explorer functions will always return a new dataframe or series.

Links:

Acknowledgements

Explorer is an extensive library and there’s much more we could say. But for now, we’d just like to thank the dozens of contributors who’ve added wonderful improvements over the years. :heart:

16 comments
#explorer
58 1448 16

Most Liked

billylanchantin

billylanchantin

Explorer - version 0.8

Explorer has released version 0.8!

Added

  • Add explode/2 to Explorer.DataFrame. This function is useful to expand the contents of a {:list, inner_dtype} series into a “inner_dtype” series.

  • Add the new series functions all?/1 and any?/1, to work with boolean series.

  • Add support for the “struct” dtype. This new dtype represents the struct dtype from Polars/Arrow.

  • Add map/2 and map_with/2 to the Explorer.Series module.
    This change enables the usage of the Explore.Query features in a series.

  • Add sort_by/2 and sort_with/2 to the Explorer.Series module.
    This change enables the usage of the lazy computations and the Explorer.Query module.

  • Add unnest/2 to Explorer.DataFrame. It works by taking the fields of a “struct” - the new dtype - and transform them into columns.

  • Add pairwise correlation - Explorer.DataFrame.correlation/2 - to calculate the correlation between numeric columns inside a data frame.

  • Add pairwise covariance - Explorer.DataFrame.covariance/2 - to calculate the covariance between numeric columns inside a data frame.

  • Add support for more integer dtypes. This change introduces new signed and unsigned integer dtypes:

    • {:s, 8}, {:s, 16}, {:s, 32}
    • {:u, 8}, {:u, 16}, {:u, 32}, {:u, 64}.

    The existing :integer dtype is now represented as {:s, 64}, and it’s still the default dtype for integers. But series and data frames can now work with the new dtypes. Short names for these new dtypes can be used in functions like Explorer.Series.from_list/2. For example, {:u, 32} can be represented with the atom :u32.

    This may bring more interoperability with Nx, and with Arrow related things, like ADBC and Parquet.

  • Add ewm_standard_deviation/2 and ewm_variance/2 to Explorer.Series.
    They calculate the “exponentially weighted moving” variance and standard deviation.

  • Add support for :skip_rows_after_header option for the CSV reader functions.

  • Support {:list, numeric_dtype} for Explorer.Series.frequencies/1.

  • Support pins in cond, inside the context of Explorer.Query.

  • Introduce the :null dtype. This is a special dtype from Polars and Apache Arrow to represent “all null” series.

  • Add Explorer.DataFrame.transpose/2 to transpose a data frame.

Changed

  • Rename the functions related to sorting/arranging of the Explorer.DataFrame.
    Now arrange_with is named sort_with, and arrange is sort_by.

    The sort_by/3 is a macro and it is going to work using the Explorer.Query module. On the other side, the sort_with/2 uses a callback function.

  • Remove unnecessary casts to {:s, 64} now that we support more integer dtypes.
    It affects some functions, like the following in the Explorer.Series module:

    • argsort
    • count
    • rank
    • day_of_week, day_of_year, week_of_year, month, year, hour, minute, second
    • abs
    • clip
    • lengths
    • slice
    • n_distinct
    • frequencies

    And also some functions from the Explorer.DataFrame module:

    • mutate - mostly because of series changes
    • summarise - mostly because of series changes
    • slice

Fixed

  • Fix inspection of series and data frames between nodes.

  • Fix cast of :string series to {:datetime, any()}

  • Fix mismatched types in Explorer.Series.pow/2, making it more consistent.

  • Normalize sorting options.

  • Fix functions with dtype mismatching the result from Polars.
    This fix is affecting the following functions:

    • quantile/2 in the context of a lazy series
    • mode/1 inside a summarisation
    • strftime/2 in the context of a lazy series
    • mutate_with/2 when creating a column from a NaiveDateTime or Explorer.Duration.

Contributors

Thank you to everyone who opend up a PR:

And thank you to the first-time contributors!:

Changelogs

Full Changelog: Comparing v0.7.2...v0.8.0 · elixir-explorer/explorer · GitHub
Official Changelog: Changelog — Explorer v0.11.1

10
Post #2
billylanchantin

billylanchantin

[Release] Explorer - v0.10

Added

  • Add support for the decimals data type.

    Decimals dtypes are represented by the {:decimal, precision, scale} tuple,
    where precision can be a positive integer from 0 to 38, and is the maximum number
    of digits that can be represented by the decimal. The scale is the number of
    digits after the decimal point.

    With this addition, we also added the :decimal package as a new dependency.
    The Explorer.Series.from_list/2 function accepts decimal numbers from that
    package as values * %Decimal{}.

    This version has a small number of operations, but is a good foundation.

  • Allow the usage of queries and lazy series outside callbacks and macros.
    This is an improvement to functions that were originally designed to accept callbacks.
    With this change you can now reuse lazy series across different “queries”.
    See the Explorer.Query docs for details.

    The affected functions are:

    • Explorer.DataFrame.filter_with/2
    • Explorer.DataFrame.mutate_with/2
    • Explorer.DataFrame.sort_with/2
    • Explorer.DataFrame.summarise_with/2

  • Allow accessing the dataframe inside query.

  • Add “lazy read” support for Parquet and NDJSON from HTTP(s).

  • Expose more options for Explorer.Series.cut/3 and Explorer.Series.qcut/3.
    These options were available in Polars, but not in our APIs.

Fixed

  • Fix creation of series where a nil value inside a list * for a {:list, any()} dtype -
    could result in an incompatible dtype. This fix will prevent panics for list of lists with
    nil entries.

  • Fix Explorer.DataFrame.dump_ndjson/2 when date time is in use.

  • Fix Explorer.Series.product/1 for lazy series.

  • Accept %FSS.HTTP.Entry{} structs in functions like Explorer.DataFrame.from_parquet/2.

  • Fix encode of binaries to terms from series of the {:struct, any()} dtype.
    In case the inner fields of the struct had any binary (:binary dtype), it was
    causing a panic.

Changed

  • Change the defaults of the functions Explorer.Series.cut/3 and Explorer.Series.qcut/3
    to not have “break points” column in the resultant dataframe.
    So the :include_breaks is now false by default.

Contributors

New Contributors

Changelogs

8
Post #5
billylanchantin

billylanchantin

[Blog] Explorer 0.8: The dtype release

5
Post #3
billylanchantin

billylanchantin

[Release] Explorer - v0.11.0

Version 0.11.0 has been released!

This one is mostly minor improvements and bugfixes. One notable change is that the DataFrame print format was altered to save vertical space and to hint when rows were hidden.

Before:

iex> Explorer.Datasets.iris() |> Explorer.DataFrame.print()
+-----------------------------------------------------------------------+
|              Explorer DataFrame: [rows: 150, columns: 5]              |
+--------------+-------------+--------------+-------------+-------------+
| sepal_length | sepal_width | petal_length | petal_width |   species   |
|    <f64>     |    <f64>    |    <f64>     |    <f64>    |  <string>   |
+==============+=============+==============+=============+=============+
| 5.1          | 3.5         | 1.4          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 4.9          | 3.0         | 1.4          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 4.7          | 3.2         | 1.3          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 4.6          | 3.1         | 1.5          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 5.0          | 3.6         | 1.4          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+

After:

iex> Explorer.Datasets.iris() |> Explorer.DataFrame.print()
+--------------------------------------------------------------------------+
|               Explorer DataFrame: [rows: 150, columns: 5]                |
+--------------+-------------+--------------+-------------+----------------+
| sepal_length | sepal_width | petal_length | petal_width |    species     |
|    <f64>     |    <f64>    |    <f64>     |    <f64>    |    <string>    |
+==============+=============+==============+=============+================+
| 5.1          | 3.5         | 1.4          | 0.2         | Iris-setosa    |
| 4.9          | 3.0         | 1.4          | 0.2         | Iris-setosa    |
| 4.7          | 3.2         | 1.3          | 0.2         | Iris-setosa    |
| …            | …           | …            | …           | …              |
| 6.2          | 3.4         | 5.4          | 2.3         | Iris-virginica |
| 5.9          | 3.0         | 5.1          | 1.8         | Iris-virginica |
+--------------+-------------+--------------+-------------+----------------+

See here for details on the new format:

Added

  • Explorer.DataFrame.estimated_size/1 - Estimates memory size of a DataFrame
  • Explorer.DataFrame.to_table_string/2 - Represents a DataFrame as a string
    for printing
  • Explorer.Series.degrees/1 - Converts radians to degrees
  • Explorer.Series.radians/1 - Converts degrees to radians
  • :quote_style option to CSV functions

Fixed

  • Fix bug where :region was incorrectly required in %FSS.S3.Entry{}
  • Fix trigonometric functions to not raise on f32
  • Fix warning from :table_rex dependency when printing
  • Fix formatting of Explorer.DataFrame.mutate_with/2 options
  • Explorer.Series.fill_missing/2 now works for all integer and float dtypes
  • Explorer.Series.frequencies/1 now works for {:list, _} dtype
  • Fix inefficiency with categorization
  • Fix typespecs
    • Explorer.DataFrame.select/2
    • Explorer.DataFrame.ungroup/1
    • Explorer.Series functions that may return lazy series

Changed

  • Printing a DataFrame looks different
    • Adds a row of … to indicate there are hidden rows. Includes a new option
      limit_dots: :bottom | :split to specify how to do this.
    • Drops the row separators except when composite dtypes are present.
    • Allows you to pass through valid options to TableRex.render!/2. This
      gives you a little more flexibility in case you don’t like the defaults.
  • Explorer.DataFrame.print/1 now documents its default :limit of 5 rows
  • Explorer.DataFrame.concat_rows/1 has improved error messages
  • Accessing a DataFrame with a range now raises if the range is out of bounds

New Contributors

Full Changelog

The full changelog includes all contributions with individual attributions. Special shoutout to @mhanberg for helping out with some gnarly version wrangling!

4
Post #13
billylanchantin

billylanchantin

[Release] Explorer - v0.11.1

Version 0.11.1 released!

This is a small one, mostly because we broke printing for lazy dataframes :grimacing: (thanks for the patch, @mhanberg!). But it comes with a few improvements too.

Of particular note is that Explorer.DataFrame.group_by is no longer stable by default. Before, groups would always be returned in their original order which is nice but has a non-trivial performance penalty. Now groups are returned in a random order for a performance boost (thanks, @petrkozorezov!). But you can always set group_by(..., stable: true) if stability is needed.

Added

  • Explorer.DataFrame.dump_ipc_schema
  • Explorer.DataFrame.dump_ipc_record_batch
  • Explorer.Series.cumulative_count
  • :stable option for Explorer.DataFrame.group_by

Fixed

  • Fix printing lazy data frame with new default print options (as of v0.11.0)
  • Fix mutate docs formatting

New Contributors

Full Changelog

The full changelog includes all contributions with individual attributions.

4
Post #14

Where Next?

View thread on forum (has 16 responses!)
explorer
Home News>Announcing
#explorer
58 1449 16
Last post

Popular in Announcing Top

zachdaniel
News>Announcing

AshEvents: Event Sourcing Made Simple For Ash
Hey folks! AshEvents Release We’ve just released the first version of AshEvents, an Event Sourcing tool for Ash Framework apps. Check o...
/ash#ash-news#ash-extension
137 1520 44
New
martosaur
News>Announcing

InstructorLite – structured outputs for LLMs, a tinkering-friendly fork of Instructor
Hello, I’m excited to introduce InstructorLite – a fork of the Instructor package. Instructor brought the very idea of structured LLM p...
#instructor_lite
39 970 10
New
sorentwo
News>Announcing

Oban - Reliable and Observable Job Processing
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
/oban#ecto#postgres#job-processing
977 41022 311
New
RobinBoers
News>Announcing

Jinja - Elixir API for Jinja templates using Pythonx
Renders Jinja templates, either from disk or defined at runtime, with support for all native Jinja features (including extends). It does...
#python#templates
8 197 3
New
rodloboz
News>Announcing

Lotus - Lightweight, SQL query runner and storage for Phoenix apps
I’ve started working on a new library to run SQL queries and do basic business intelligence. Think “Blazer for Elixir.” Currently it fe...
#ecto#postgresql#sqlite#lotus
87 952 23
New
cheekyfinder
News>Announcing

ExSift - High-performance MongoDB-style query filtering for Elixir
ExSift: High-performance MongoDB-style query filtering for Elixir I’m working on ExSift, a high-performance library that uses sift.js-ins...
#data-structures#mongodb#learning-elixir#filtering
2 113 1
New
lud
News>Announcing

Oaskit - OpenAPI 3.1 based validation for Phoenix
Hello! I’ve been working on the Oaskit library for a while now, and just released a first version. Since I’ve built JSV I wanted to be ...
/phoenix#openapi#oaskit
29 419 9
New
type1fool
News>Announcing

WebAuthnLiveComponent - Passwordless Auth for LiveView Apps
WebAuthnLiveComponent WebAuthnComponents See this post about renaming the package. Passwordless authentication for Phoenix LiveView app...
/phoenix#authentication#security#liveview
214 6960 49
New
leandrocp
News>Announcing

MDEx - Fast and Extensible Markdown
MDEx is a fast and extensible Markdown parser and formatter. Fast Leverage Rust to parse, manipulate and render documents using: comra...
#markdown#rustler#commonmark
69 1038 15
New
zoedsoupe
News>Announcing

Anubis (formerly Hermes) MCP - Model Context Protocol Implementation for Elixir
update (since 2025-07-24) the project got forked and rebranded to anubis-mcp, since i not on CloudWalk anymore and can’t ensure they will...
#library#mcp
44 1908 25
New

Other popular topics Top

stefanchrobot
Questions & Help>Questions

How to decode a JSON into a struct safely?
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
#structs#json
29 20780 26
New
sorentwo
News>Announcing

Oban - Reliable and Observable Job Processing
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
/oban#ecto#postgres#job-processing
977 41022 311
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
polypush135
Questions & Help>Questions

Proper use of Task.async?
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
#task
26 15568 8
New
nsuchy
Questions & Help>Questions

IEX in Windows Powershell?
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
#iex#microsoft-windows#windows#powershell
15 29309 4
New
vac
Questions & Help>Questions

Formatting a string to PEM with headers and line breaks
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
2 14555 7
New
mgjohns61585
Questions & Help>Questions

Convert a string to an integer?
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
#how-to-question
3 14200 6
New
stefanluptak
Questions & Help>Questions

Using VSCode on multiple monitors
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
#vscode#vscode-elixir
7 17615 6
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
#ecto
22 17588 9
New
Qqwy
Questions & Help>Questions

Updating structs: Map.put vs %Foo{oldfoo | new: value} vs put_in
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
#maps#structs#pipeline#access
115 28410 31
New
Back to top

News>Announcing

News Announcing ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

WyewWorks

Enabling companies to succeed by building software people love.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement