bgpratt2000

bgpratt2000

Explorer.DataFrame and Livebook help

Hello!

I am a BEGINNER and apologize in advance for the elementary questions.

I want to do some data transformations to a dataframe in Livebook, but currently one of the dataframe’s columns/series representing a :utc_datetime from ecto has a :string dtype, and I cannot find a simple/automated way to convert this from a :string to a :utc_datetime and then to an Explorer.Series :date.

I followed this procedure to set up the livebook:

  1. ran an ecto.query on my postgresql database, where my select statement returns a map with atom keys
  2. passed the query as an argument to Ecto.Repo.all()
  3. converted the map from Repo.all() to a csv file
  4. so far have been doing useful data transforms and visualization using Explorer and Vega_Lite in Livebook, HOWEVER: in livebook the function Explorer.DataFrame.from_csv!/1 interpreted a :utc_datetime column as a :string series, and I don’t know how to convert the :string back to any useful/date-related Explorer.Series dtype.

At this exact moment I would appreciate a quick fix to this conversion struggle, but in general I hope to soon use livebook for more rigorous machine learning algorithms and data manipulations on data from my db, so any and all ways to streamline the above clunky process would be GREATLY appreciated.

Thanks! Y’all are awesome!!!

Marked As Solved

dtew

dtew

The following is Claude’s LLM answer to your query - hope it helps you!

Here’s how you can handle this:

  1. When loading the CSV, you can use the parse_dates option to automatically parse date/datetime strings:
df = Explorer.DataFrame.from_csv!("your_file.csv", parse_dates: true)

If that doesn’t work automatically, you can manually convert the column using mutate. Here’s how:

require Explorer.DataFrame

df = Explorer.DataFrame.mutate(
  df,
  datetime_col: cast(string_datetime_col, :datetime)
)

For more control over the parsing, you can use Explorer’s datetime casting functions. Here’s a complete example:

require Explorer.DataFrame, as: DF

# Starting with your string datetime column
df = DF.from_csv!("your_file.csv")

# Convert string to datetime
df = DF.mutate(
  df,
  datetime_col: strptime(string_datetime_col, "%Y-%m-%d %H:%M:%S", unit: :microsecond)
)

# If you specifically need just the date portion:
df = DF.mutate(
  df,
  date_col: dt_date(datetime_col)
)

For a more streamlined workflow from Ecto to Explorer, consider:

  1. Use Ecto’s select_merge to format dates in the query itself
  2. Or use Explorer’s ADBC functionality to read directly from your database:
{:ok, df} = Explorer.DataFrame.from_query(
  conn,  # ADBC connection
  "SELECT * FROM your_table",
  []
)

This would give you properly typed columns directly from your database without needing the CSV intermediate step.

The key functions to remember for datetime manipulation in Explorer are:

  • strptime - Parse string to datetime
  • cast - General type conversion
  • dt_date - Extract date from datetime
  • parse_dates option in from_csv

Where Next?

Popular in Questions Top

lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

Other popular topics Top

TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

We're in Beta

About us Mission Statement