That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton
Most of these frameworks now have docs / plugins / sister libraries geared around AI agents
It would be really helpful to read a good technical blog doing a landscape of design patterns in these different approaches, and thoughts on how to fit things together well into a pipeline given various quirks of LLMs (e.g. nondeterminism).
This page is a good start, even if it is written as an airflow-specific how-to!
Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious.
Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe 2gb out of 8gb system RAM in use. No reason is given in the logs.
I am trying to switch to dagster, not because it's awesome, but because it hasn't crashed randomly on me.
You're right, it doesn't happen when developing locally, only in MWAA. This was the answer given by the Airflow team as well and I figured they would punt before I asked.
I realize Amazon is taking an open source project and making a ton of money on it (the instance prices are ridiculous for what you get) and the incentives are misaligned for the Airflow team to help AWS make it better unless AWS paid them to help fix it.
It's crap all around, and Airflow gets a bad rap from AWS's terrible MWAA product based on it.
Their operational perspective is catastrophic; how does one view the logs for a dag through the UI[1]? Why can't it store the python in the database they have attached to their deployment, versus making me jump through 80,000 hoops to put the files in the right magic directory on disk of every worker[2]?
We deploy on K8s in OpenStack from a scheduled GitHub Actions pipeline which aggregates DAGs into a new container build based on hashes of hashes. This works well with almost no intervention.
WRT your 1, above any DAG output to stdout/err is available via the logs tab from the graph view of the individual tasks. Almost all our DAGs leverage on the PythonOperator though, not sure if that standardises this for us and your experience is muddied by more complexity than we currently have?
WRT 2. we generate an uber requirements.txt running pyreqs from the pipeline and install everything in the container automatically. Again no issues currently - although we do need to manually add the installation of test libraries to the pipeline job as for some reason auto-discovery is flakier for unit-tests frameworks.
Prefect is amazing. Built out an ETL pipeline system with it at last job and would love to get it incorporated in the current one, but unfortunately have a lot of legacy stuff in Airflow. Being able to debug stuff locally was amazing and super clean integration with K8S.
Other guy said it right. These work and are fine but you lose the legacy stuff. If you know your limits and where the eventual system will end up it's great and probably better.
If you are building a expandable long term system and you want all the goodies baked in choose airflow.
Pretty much the same as any architecture choice. Ugly/hard often means control and features, pretty/easy means less of both.
On the surface the differences are not very noticable other than the learning curve of getting started.
+1 to this. other solutions over-promise, under-deliver, poor developer relations and communication, "open-source, but pay us" style open-source, and is indeed a mess
That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton
Most of these frameworks now have docs / plugins / sister libraries geared around AI agents
It would be really helpful to read a good technical blog doing a landscape of design patterns in these different approaches, and thoughts on how to fit things together well into a pipeline given various quirks of LLMs (e.g. nondeterminism).
This page is a good start, even if it is written as an airflow-specific how-to!