Lots of asks about an Airflow charm.
Thought I would start the conversation about what a good path forward may be.
1 Like
I think there is a fair bit of space to figure out what would Airflow as-a-charm actually going to function. Is the intent that you would just deploy it as infrastructure, or is the intent that you would deploy the infrastructure as charms and have them register themselves to Airflow.
I’m curious what the actual requests are.
1 Like
Our primary use case is to have airflow run spark-submit based pipelines.
1 Like
On the face of it, it would make the most sense to me if other services that are provisioned as charms (databases, file systems, …) could somehow register themselves with Airflow. Airflow wants (at least) read access to as many services as possible.
One of the tricky things to work out is how secrets should be managed. Airflow wants to have control over secrets itself, but I think that a Juju deployment would like everything in code.
Using juju, totally. The only dependency that really needs to be local to airflow is the spark-submit executable.
Right, use juju for the service discovery and charm code to handle writing out the correct config for each service. This seems dead on.