This is a rather fuzzy question but… is there any predominant or favorite ETL or ELT pipeline setup in juju ?? Reason I ask is… I am about to embark likely on writing some charms around Singer… Meltano… and/or Apache Airflow…
The gist is… I have a PostGresql DB… which I want to fill with data in a more professional production grade way than a simple CRON job executing python
any opinions here from fellow charmers? or am I wading into territory that’s green?
@ruivasconcelos1 Has the genesis of an operator for Airflow as part of an internal charming campaign at Canonical. You can find the code here.
There has been talk of using Airflow more widely within Canonical so I anticipate there will be attention given to this over time in terms of completeness and quality.
Perhaps you can help move it forward.
Edit: I actually meant to tag @jfguedez there, who is the author.
I have to confess I’ve avoided K8s based charms… but I can see the writing on the wall that a LOT of work is going on there. I will spin up Meltano as best I can… and maybe once I get comfortable I can inspect if I should go K8s for all my charms… so as to help test/use that airflow implementation
EDIT: It turns out Meltano has the option to install airflow within it
https://meltano.com/docs/orchestration.html#installing-airflow
So I will charm meltano first… use the built-in airflow… and see how far that gets me. Thanks @manadart for the heads up
