News>Announcing

Explorer - Series (1D) and dataframes (2D) for fast and elegant data exploration in Elixir

#explorer
billylanchantin

billylanchantin

Explorer - Series (1D) and dataframes (2D) for fast and elegant data exploration in Elixir

The Explorer library has been out for some time. But we just released the latest version (see below), and we thought we’d start posting updates to the forum.

Wait, Explorer is new to me. What are Series and DataFrames?

Explorer is a DataFrame library for Elixir.

DataFrame libraries are common in languages which have a focus on data manipulation, including:

If you’d like a more in-depth tutorial, there’s an excellent LiveBook called Ten Minutes to Explorer that you can play with:

But we’ll provide a quick overview here.

Briefly, you can think of a DataFrame like an in-memory table. Its purpose is to facilitate common data exploration and analysis tasks. As such, it’s a column-oriented table.

Column-oriented tables

If you’re unfamiliar with column-oriented tables, suppose you have a table of pet data like this:

type age color
cat 5 black
dog 2 brown
dog 3 brindle

A row-oriented organization of that data might look like this in Elixir:

rows = [
  [type: "cat", age: 5, color: "black"],
  [type: "dog", age: 2, color: "brown"],
  [type: "dog", age: 3, color: "brindle"],
]

It matches the original table fairly one-to-one. But the column-oriented version might instead look like:

columns = [
  type: ["cat", "dog", "dog"],
  age: [5, 2, 3],
  color: ["black", "brown", "brindle"]
]

It has same information, but “transposed”.

Column-orientation is beneficial if you’re asking questions that require a lot of number-crunching like “What’s the average age of all pets?”. In the row-oriented version, finding the average age would require first looking through the entire contents of the table to collect the relevant data. But in the column-oriented version, those values have already been co-located in memory.

Series and DataFrames: columns and tables

In dataframe parlance, a “series” is a single column and a “dataframe” is a collection of named series, aka a table.

Our example above would look like this:

type = Explorer.Series.from_list(["cat", "dog", "dog"])
age = Explorer.Series.from_list([5, 2, 3])
color = Explorer.Series.from_list(["black", "brown", "brindle"])

df = Explorer.DataFrame.new(type: type, age: age, color: color)
# #Explorer.DataFrame<
#   Polars[3 x 3]
#   type string ["cat", "dog", "dog"]
#   age s64 [5, 2, 3]
#   color string ["black", "brown", "brindle"]
# >

Some things to note:

  • Each series has a corresponding data type or “dtype”, e.g. type has the dtype string.
  • The word “Polars” appears. That indicates that this dataframe is using the backend powered by the fantastic Polars library (the default backend).

And if we really did want to know the average age of the pets, that would look like this:

Explorer.Series.mean(df["age"])
# 3.3333333333333335

Features and design

Preiminaries out of the way, here are Explorer’s high-level features:

  • Simply typed series: :binary, :boolean, :category, :date, :datetime, :duration, floats of 32 and 64 bits ({:f, size}), integers of 8, 16, 32 and 64 bits ({:s, size}, {:u, size}), :null, :string, :time, :list, and :struct.

  • A powerful but constrained and opinionated API, so you spend less time looking for the right function and more time doing data manipulation.

  • Support for CSV, Parquet, NDJSON, and Arrow IPC formats

  • Integration with external databases via ADBC and direct connection to file storages such as S3

  • Pluggable backends, providing a uniform API whether you’re working in-memory or (forthcoming) on remote databases or even Spark dataframes.

  • The first (and default) backend is based on NIF bindings to the blazing-fast polars library.

The API is heavily influenced by Tidy Data and borrows much of its design from dplyr.
The philosophy is heavily influenced by this passage from dplyr’s documentation:

  • By constraining your options, it helps you think about your data manipulation challenges.

  • It provides simple “verbs”, functions that correspond to the most common data manipulation tasks, to help you translate your thoughts into code.

  • It uses efficient backends, so you spend less time waiting for the computer.

The aim here isn’t to have the fastest dataframe library around (though it certainly helps that we’re building on Polars, one of the fastest).
Instead, we’re aiming to bridge the best of many worlds:

  • the elegance of dplyr
  • the speed of polars
  • the joy of Elixir

That means you can expect the guiding principles to be ‘Elixir-ish’. For example, you won’t see the underlying data mutated, even if that’s the most efficient implementation. Explorer functions will always return a new dataframe or series.

Links:

Acknowledgements

Explorer is an extensive library and there’s much more we could say. But for now, we’d just like to thank the dozens of contributors who’ve added wonderful improvements over the years. :heart:

16 comments
#explorer
58 1448 16

Most Liked

billylanchantin

billylanchantin

Explorer - version 0.8

Explorer has released version 0.8!

Added

  • Add explode/2 to Explorer.DataFrame. This function is useful to expand the contents of a {:list, inner_dtype} series into a “inner_dtype” series.

  • Add the new series functions all?/1 and any?/1, to work with boolean series.

  • Add support for the “struct” dtype. This new dtype represents the struct dtype from Polars/Arrow.

  • Add map/2 and map_with/2 to the Explorer.Series module.
    This change enables the usage of the Explore.Query features in a series.

  • Add sort_by/2 and sort_with/2 to the Explorer.Series module.
    This change enables the usage of the lazy computations and the Explorer.Query module.

  • Add unnest/2 to Explorer.DataFrame. It works by taking the fields of a “struct” - the new dtype - and transform them into columns.

  • Add pairwise correlation - Explorer.DataFrame.correlation/2 - to calculate the correlation between numeric columns inside a data frame.

  • Add pairwise covariance - Explorer.DataFrame.covariance/2 - to calculate the covariance between numeric columns inside a data frame.

  • Add support for more integer dtypes. This change introduces new signed and unsigned integer dtypes:

    • {:s, 8}, {:s, 16}, {:s, 32}
    • {:u, 8}, {:u, 16}, {:u, 32}, {:u, 64}.

    The existing :integer dtype is now represented as {:s, 64}, and it’s still the default dtype for integers. But series and data frames can now work with the new dtypes. Short names for these new dtypes can be used in functions like Explorer.Series.from_list/2. For example, {:u, 32} can be represented with the atom :u32.

    This may bring more interoperability with Nx, and with Arrow related things, like ADBC and Parquet.

  • Add ewm_standard_deviation/2 and ewm_variance/2 to Explorer.Series.
    They calculate the “exponentially weighted moving” variance and standard deviation.

  • Add support for :skip_rows_after_header option for the CSV reader functions.

  • Support {:list, numeric_dtype} for Explorer.Series.frequencies/1.

  • Support pins in cond, inside the context of Explorer.Query.

  • Introduce the :null dtype. This is a special dtype from Polars and Apache Arrow to represent “all null” series.

  • Add Explorer.DataFrame.transpose/2 to transpose a data frame.

Changed

  • Rename the functions related to sorting/arranging of the Explorer.DataFrame.
    Now arrange_with is named sort_with, and arrange is sort_by.

    The sort_by/3 is a macro and it is going to work using the Explorer.Query module. On the other side, the sort_with/2 uses a callback function.

  • Remove unnecessary casts to {:s, 64} now that we support more integer dtypes.
    It affects some functions, like the following in the Explorer.Series module:

    • argsort
    • count
    • rank
    • day_of_week, day_of_year, week_of_year, month, year, hour, minute, second
    • abs
    • clip
    • lengths
    • slice
    • n_distinct
    • frequencies

    And also some functions from the Explorer.DataFrame module:

    • mutate - mostly because of series changes
    • summarise - mostly because of series changes
    • slice

Fixed

  • Fix inspection of series and data frames between nodes.

  • Fix cast of :string series to {:datetime, any()}

  • Fix mismatched types in Explorer.Series.pow/2, making it more consistent.

  • Normalize sorting options.

  • Fix functions with dtype mismatching the result from Polars.
    This fix is affecting the following functions:

    • quantile/2 in the context of a lazy series
    • mode/1 inside a summarisation
    • strftime/2 in the context of a lazy series
    • mutate_with/2 when creating a column from a NaiveDateTime or Explorer.Duration.

Contributors

Thank you to everyone who opend up a PR:

And thank you to the first-time contributors!:

Changelogs

Full Changelog: Comparing v0.7.2...v0.8.0 · elixir-explorer/explorer · GitHub
Official Changelog: Changelog — Explorer v0.11.1

10
Post #2
billylanchantin

billylanchantin

[Release] Explorer - v0.10

Added

  • Add support for the decimals data type.

    Decimals dtypes are represented by the {:decimal, precision, scale} tuple,
    where precision can be a positive integer from 0 to 38, and is the maximum number
    of digits that can be represented by the decimal. The scale is the number of
    digits after the decimal point.

    With this addition, we also added the :decimal package as a new dependency.
    The Explorer.Series.from_list/2 function accepts decimal numbers from that
    package as values * %Decimal{}.

    This version has a small number of operations, but is a good foundation.

  • Allow the usage of queries and lazy series outside callbacks and macros.
    This is an improvement to functions that were originally designed to accept callbacks.
    With this change you can now reuse lazy series across different “queries”.
    See the Explorer.Query docs for details.

    The affected functions are:

    • Explorer.DataFrame.filter_with/2
    • Explorer.DataFrame.mutate_with/2
    • Explorer.DataFrame.sort_with/2
    • Explorer.DataFrame.summarise_with/2

  • Allow accessing the dataframe inside query.

  • Add “lazy read” support for Parquet and NDJSON from HTTP(s).

  • Expose more options for Explorer.Series.cut/3 and Explorer.Series.qcut/3.
    These options were available in Polars, but not in our APIs.

Fixed

  • Fix creation of series where a nil value inside a list * for a {:list, any()} dtype -
    could result in an incompatible dtype. This fix will prevent panics for list of lists with
    nil entries.

  • Fix Explorer.DataFrame.dump_ndjson/2 when date time is in use.

  • Fix Explorer.Series.product/1 for lazy series.

  • Accept %FSS.HTTP.Entry{} structs in functions like Explorer.DataFrame.from_parquet/2.

  • Fix encode of binaries to terms from series of the {:struct, any()} dtype.
    In case the inner fields of the struct had any binary (:binary dtype), it was
    causing a panic.

Changed

  • Change the defaults of the functions Explorer.Series.cut/3 and Explorer.Series.qcut/3
    to not have “break points” column in the resultant dataframe.
    So the :include_breaks is now false by default.

Contributors

New Contributors

Changelogs

8
Post #5
billylanchantin

billylanchantin

[Blog] Explorer 0.8: The dtype release

5
Post #3
billylanchantin

billylanchantin

[Release] Explorer - v0.11.0

Version 0.11.0 has been released!

This one is mostly minor improvements and bugfixes. One notable change is that the DataFrame print format was altered to save vertical space and to hint when rows were hidden.

Before:

iex> Explorer.Datasets.iris() |> Explorer.DataFrame.print()
+-----------------------------------------------------------------------+
|              Explorer DataFrame: [rows: 150, columns: 5]              |
+--------------+-------------+--------------+-------------+-------------+
| sepal_length | sepal_width | petal_length | petal_width |   species   |
|    <f64>     |    <f64>    |    <f64>     |    <f64>    |  <string>   |
+==============+=============+==============+=============+=============+
| 5.1          | 3.5         | 1.4          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 4.9          | 3.0         | 1.4          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 4.7          | 3.2         | 1.3          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 4.6          | 3.1         | 1.5          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+
| 5.0          | 3.6         | 1.4          | 0.2         | Iris-setosa |
+--------------+-------------+--------------+-------------+-------------+

After:

iex> Explorer.Datasets.iris() |> Explorer.DataFrame.print()
+--------------------------------------------------------------------------+
|               Explorer DataFrame: [rows: 150, columns: 5]                |
+--------------+-------------+--------------+-------------+----------------+
| sepal_length | sepal_width | petal_length | petal_width |    species     |
|    <f64>     |    <f64>    |    <f64>     |    <f64>    |    <string>    |
+==============+=============+==============+=============+================+
| 5.1          | 3.5         | 1.4          | 0.2         | Iris-setosa    |
| 4.9          | 3.0         | 1.4          | 0.2         | Iris-setosa    |
| 4.7          | 3.2         | 1.3          | 0.2         | Iris-setosa    |
| …            | …           | …            | …           | …              |
| 6.2          | 3.4         | 5.4          | 2.3         | Iris-virginica |
| 5.9          | 3.0         | 5.1          | 1.8         | Iris-virginica |
+--------------+-------------+--------------+-------------+----------------+

See here for details on the new format:

Added

  • Explorer.DataFrame.estimated_size/1 - Estimates memory size of a DataFrame
  • Explorer.DataFrame.to_table_string/2 - Represents a DataFrame as a string
    for printing
  • Explorer.Series.degrees/1 - Converts radians to degrees
  • Explorer.Series.radians/1 - Converts degrees to radians
  • :quote_style option to CSV functions

Fixed

  • Fix bug where :region was incorrectly required in %FSS.S3.Entry{}
  • Fix trigonometric functions to not raise on f32
  • Fix warning from :table_rex dependency when printing
  • Fix formatting of Explorer.DataFrame.mutate_with/2 options
  • Explorer.Series.fill_missing/2 now works for all integer and float dtypes
  • Explorer.Series.frequencies/1 now works for {:list, _} dtype
  • Fix inefficiency with categorization
  • Fix typespecs
    • Explorer.DataFrame.select/2
    • Explorer.DataFrame.ungroup/1
    • Explorer.Series functions that may return lazy series

Changed

  • Printing a DataFrame looks different
    • Adds a row of … to indicate there are hidden rows. Includes a new option
      limit_dots: :bottom | :split to specify how to do this.
    • Drops the row separators except when composite dtypes are present.
    • Allows you to pass through valid options to TableRex.render!/2. This
      gives you a little more flexibility in case you don’t like the defaults.
  • Explorer.DataFrame.print/1 now documents its default :limit of 5 rows
  • Explorer.DataFrame.concat_rows/1 has improved error messages
  • Accessing a DataFrame with a range now raises if the range is out of bounds

New Contributors

Full Changelog

The full changelog includes all contributions with individual attributions. Special shoutout to @mhanberg for helping out with some gnarly version wrangling!

4
Post #13
billylanchantin

billylanchantin

[Release] Explorer - v0.11.1

Version 0.11.1 released!

This is a small one, mostly because we broke printing for lazy dataframes :grimacing: (thanks for the patch, @mhanberg!). But it comes with a few improvements too.

Of particular note is that Explorer.DataFrame.group_by is no longer stable by default. Before, groups would always be returned in their original order which is nice but has a non-trivial performance penalty. Now groups are returned in a random order for a performance boost (thanks, @petrkozorezov!). But you can always set group_by(..., stable: true) if stability is needed.

Added

  • Explorer.DataFrame.dump_ipc_schema
  • Explorer.DataFrame.dump_ipc_record_batch
  • Explorer.Series.cumulative_count
  • :stable option for Explorer.DataFrame.group_by

Fixed

  • Fix printing lazy data frame with new default print options (as of v0.11.0)
  • Fix mutate docs formatting

New Contributors

Full Changelog

The full changelog includes all contributions with individual attributions.

4
Post #14

Where Next?

View thread on forum (has 16 responses!)
explorer
Home News>Announcing
#explorer
58 1449 16
Last post

Popular in Announcing Top

BartOtten
News>Announcing

Routex - build powerful Phoenix routes: localize, customize, and innovate
This powerful library works together with Phoenix Router to provide the ultimate routing solution. It simplifies route manipulation, givi...
/phoenix#i18n#liveview#internationalization#routing#routex
80 1255 10
New
mspanc
News>Announcing

Membrane - a new framework for multimedia processing
I am pleased to announce an initial release of the Membrane Framework - an Elixir-based framework with special focus on processing multim...
/membrane#multimedia
164 10885 49
New
jarlah
News>Announcing

Testcontainers - A Testcontainers implementation for Elixir
Testcontainers Testcontainers is an Elixir library that supports ExUnit tests, providing lightweight, throwaway instances of common datab...
#ecto#docker#testing#exunit
56 2484 39
New
thiagoesteves
News>Announcing

Observer Web - A self-contained liveview library to monitor applications processes/ports and tracing with Erlang debugger
Hi There, We are thrilled to announce the first sub-product from the ongoing development and research of DeployEx! Inspired by the recen...
/phoenix#tracing#observer#liveview#distributed-tracing#observability#debugger
20 301 2
New
dimamik
News>Announcing

Vault - a lightweight process-scoped global data storage with immutability guarantees
Vault is a lightweight Elixir library for immutable data storage within a process subtree. Due to Elixir’s actor model nature, it’s comm...
#processes#vault#storage#context
62 951 23
New
Antrater
News>Announcing

Moon LiveView - component library
Hi there! At Moon Design System, we have been working hard for the past six months on the next generation of our LiveView component libra...
#liveview
14 491 3
New
LostKobrakai
News>Announcing

PhoenixVite - Integrating the vite built tooling with phoenix
I’ve recently created a small library phoenix_vite integrating the vite build tooling with phoenix. It provides an igniter.installer t...
/phoenix#javascript#bun#igniter
51 773 5
New
leandrocp
News>Announcing

MDEx - Fast and Extensible Markdown
MDEx is a fast and extensible Markdown parser and formatter. Fast Leverage Rust to parse, manipulate and render documents using: comra...
#markdown#rustler#commonmark
69 1038 15
New
germsvel
News>Announcing

PhoenixTest - a unified way of writing feature tests for LiveView and static pages
PhoenixTest provides a unified way of writing feature tests – regardless of whether you’re testing LiveView pages or static pages. It al...
/phoenix#ecto#testing#phoenix_test
53 1687 18
New
waseigo
News>Announcing

DoTrace - Provides a trace/1 macro to trace function calls at runtime, mimicking Clojure's dotrace output
I saw this LinkedIn post: *Can your programming language do this? This is a macro in Clojure called `dotrace`. When you surround a pie...
#metaprogramming
34 326 16
New

Other popular topics Top

JakeBecker
Chat & Discussions>Discussions

ElixirLS - the Elixir Language Server
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
#elixir-ls
1140 51847 244
New
sorentwo
News>Announcing

Oban - Reliable and Observable Job Processing
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
/oban#ecto#postgres#job-processing
977 41022 311
New
grych
Your Libraries & Projects>Libraries

Drab: remote controlled frontend framework for Phoenix
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
/phoenix#web-frameworks#frontend#javascript#drab
639 49522 488
New
quazar
Questions & Help>Questions

How to encode ecto schema with Jason with all fields
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
/phoenix#json#jason
17 13698 9
New
lk-geimfari
Questions & Help>Questions

The most correct way to read a JSON file
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
23 13679 2
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
Qqwy
Questions & Help>Questions

Updating structs: Map.put vs %Foo{oldfoo | new: value} vs put_in
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
#maps#structs#pipeline#access
115 28410 31
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
siddhant3030
Questions & Help>Questions

How can I write a raw sql query?
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
/phoenix#ecto
13 19061 20
New
magnetic
Learning Resources>Guides/Tuts/Tips/Info

Configuring VSCode to use with Elixir and Phoenix Templates
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
#phoenix_html#vs-code#vscode-elixir#elixir-ls
43 16832 15
New
Back to top

News>Announcing

News Announcing ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement