#AIRFLOW KUBERNETES WORKER HOW TO#

You have a high number of short-running tasks and want to ensure low latency between tasks.You want to use different worker types based on the type of task you're running.You're just getting started with Airflow.Specifically, the Celery executor is a good fit for your Deployment if: The Celery executor is a good option for most use cases. Astronomer uses worker autoscaling logic to determine how many workers run on your Deployment at a given time. It uses a group of workers, each of which can run multiple tasks at a time. The Celery executor is the default for all new Deployments.

#AIRFLOW KUBERNETES WORKER HOW TO#

For information about how to change the executor of an existing Deployment, see Update the Deployment executor.

Read the following topics to learn about the benefits and limitations of each executor. Astro currently supports two executors, both of which are available in the Apache Airflow open source project: The executor you choose affects the infrastructure cost of a Deployment and how efficiently your tasks execute. The difference between executors is based on how tasks are distributed across worker resources. To learn more about executors in Airflow, see Airflow executors. Use the information provided in this topic to learn how to configure the Celery and Kubernetes executors on Astro. After you choose an executor for an Astro Deployment, you can configure your DAGs and Deployment resources to maximize the executor's efficiency and performance. On Astro, every Deployment requires an executor and you can update the executor at any time. The Airflow executor determines which worker resources run your scheduled tasks.

Airflow kubernetes worker