felix-starman
Livebook as a replacement for scheduled ETL?
So I have been writing Elixir for the better part of a decade now and using Livebook to solve small problems for myself for daily operations but I am trying to encourage our data analyst to investigate it as well instead of recreating our ecto queries in SQL and then running them in bigquery which seems like a waste.
A lot of these need to be nightly scheduled queries, but I don’t want to require them to have to write oban jobs if they want just get started at doing something.
Has anyone come up with a way to run a livebook on a schedule without user input?
Marked As Solved
jonatanklosko
Hey @felix-starman, we plan to explore scheduled execution in the future, most likely in relation to Livebook Teams, tracked in Allow scheduled execution of multi-session apps · Issue #2091 · livebook-dev/livebook · GitHub.
That said, you should be able to create a notebook or app that runs code periodically. Here’s a quick idea:
<!-- livebook:{"app_settings":{"slug":"cron"}} -->
# Cron
```elixir
Mix.install([
{:kino, "~> 0.12.3"},
{:quantum, "~> 3.5"}
])
```
## Section
```elixir
defmodule Scheduler do
use Quantum, otp_app: :scheduler
end
defmodule Tasks do
require Logger
def hello() do
Logger.info("Hello #{DateTime.utc_now()}")
end
end
Application.put_env(:scheduler, Scheduler,
jobs: [
{"* * * * *", {Tasks, :hello, []}}
]
)
Kino.start_child(Scheduler)
```
In the task you could even do RPC into a remote node.
You can deploy that notebook as an app, so it is more “permanent”, and you most likely want it to run on a cloud instance. There are already options to deploy with Docker, and we are working on Livebook Teams, where you will be able to deploy an app like this into your infrastructure with a single click.
These are just some quick notes, this is definitely open to ideas and future features : )
Also Liked
jonatanklosko
Notebooks are isolated, so there is no way to share modules between them.
In the notebook you could do this:
button = Kino.Control.button("Run hello") |> Kino.render()
Kino.listen(button, fn _ -> Tasks.hello() end)
if Kino.Hub.app_info().type != :none do
Kino.start_child(Scheduler)
end
So there is an explicit button to run the task on demand, and the cron is only started when the notebook runs as an app (as opposed to just being open and edited). This way you can reuse the same notebook for both manual and scheduled. And this way it should be fine to use a single notebook that defines all the tasks, each of them can be run with a button, and there’s a single cron config for all of them.
If you want to define tasks in separate notebooks and have a single notebook with a cron, then the modules would need to be a dependency or perhaps they would live in a production node that the notebook RPCs into (which may be the way to go anyway, so that you can use all the application code/setup, but that depends on the exact use case).
larshei
I am very much interested in exactly this usecase as well.
Currently running a lot of “Pipelines” (ETL) in Azure, which are … not my personal favourite.
Looking to replace those with scripts or livebooks. We already have Livebook Teams available but unfortunately haven’t gotten to use it much within the team as of yet.
I think instead of scheduling we run a RabbitMQ consumer in each livebook and use messages to trigger runs.
That way, the livebook can be deployed permanently and only does work when required.
josevalim
Latest Livebook supports running a proxy/web server too, so you could also listen to HTTP requests and start workflows.
Popular in Questions
Other popular topics
Latest Oban Threads
Latest Livebook Threads
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"