- Managed Airflow

Working with managed airflow on GCP.

We now have a workflow tool set up for managing model workflows, but the default configuration does not provide high-availability, where new machines are provisioned when failures occur. While it’s possible to set up a distributed version of Airflow using Celery to set up different Scheduler and Worker nodes, one recent trend is using Kubernetes to create more robust Airflow deployments.

Get hands-on with 1400+ tech skills courses.