Hi all,
I’m excited to share a new Charmed Apache Airflow + Spark workshop - a reproducible, end-to-end path for deploying Charmed Apache Airflow 3.x and Charmed Apache Spark so you can run spark jobs on kubernetes straight from your DAGs.
The headline win: zero hardcoded credentials in your DAG code. Your PySpark scripts contain only business logic: all the S3 endpoints, access keys, container images, and executor settings live in the Integration Hub and are injected at submit time.
Why bother integrating the two?
-
Airflow alone has no way to process large datasets at scale; tasks run in a single container with fixed memory and no way to distribute computation.
-
Spark alone has no scheduler, no dependency graph, no job-level retries, and no visibility into whether a job ran, when, or why it failed.
Together, you get scalable, distributed data processing with reliable, observable pipeline automation.
What it does
Airflow schedules the pipeline; each spark task launches its own spark pod to do the heavy lifting. This keeps the Airflow runtime lightweight while spark handles the distributed data processing.
What’s in the box
Three demo DAGs to get you going:
tiny_spark_demo
Minimal spark-submit smoke test
spark_session_demo:
Full SparkSession job that aggregates data and writes an event log to S3 (visible in the History Server)
spark_fail_demo:
A job that fails on purpose: to show how failures surface in Airflow
Object storage is provided by MicroCeph RADOS Gateway (S3-compatible), and a justfile automates the entire deployment lifecycle.
Try it out
The workshop lives in the workflows-team-workshops repo, under airflow/airflow-spark. Head there to find the demo DAGs, Terraform config, and a justfile that automates the full deployment. Full prerequisites, commands, and architecture notes are in the workshop README.
Please reach out on Matrix and file issues or enhancement suggestions on Github.
Thanks, Janani