Hello,

I am developing an Elixir library to work with Apache Iceberg called ExIceberg. My primary focus is to handle catalog responsibilities in Elixir while delegating the reading and writing processes to the Rust implementation.

Currently, we can manage some table formats using Trino through the req_trino library. However, I want to integrate Iceberg more closely with the Elixir ecosystem.

Here is some context about Apache Iceberg:

Apache Iceberg is a table format similar to Delta Lake and Hudi. It is not responsible for computing. In Java, we have implementations that work with Spark; in Python, there is some support via PyArrow; and in Rust, it will be implemented using Apache DataFusion as the query engine.

My initial idea was to get the RecordBatch returned from the Rust implementation and pass it to Explorer. However, after discussing with @philss about the responsibilities of Explorer, I want to clarify what exactly Explorer intends to achieve.

Initially, I envisioned Explorer as similar to Ibis. But since the release of Remote DataFrame, I see that Explorer is aiming to be more like Daft.

So, I have some questions:

What is the focus of Explorer? Is it to be more of a DataFrame interface for multiple backends or a query engine with a DataFrame API?

If the focus is to be a DataFrame interface, I can return just a RecordBatch or a Table from Arrow and pass it to Explorer. For this, we might need something like this: daft.from_arrow — Daft documentation. However, if the focus is on distributed reading, ExIceberg will focus solely on catalog interaction, and Explorer will use ExIceberg to get the metadata and Explorer read the files.