Architecture overview

Charmed Airflow is an open source, cloud-native solution that helps you deploy Apache Airflow on Kubernetes using Juju and operate it throughout its lifecycle.

This document describes the layers that make it up, the role of each component, and how those components are connected.

Charmed Airflow architecture layers, from infrastructure to components.

From the diagram above:

  • Infrastructure layer. Charmed Airflow runs on any CNCF-certified Kubernetes distribution, whether on public cloud, private cloud, self-hosted infrastructure, or a local development cluster.

  • Ingress layer. Traefik (traefik-k8s) routes external traffic to the Airflow web UI and REST API and handles TLS termination (at ingress).

  • Database layer. PostgreSQL (postgresql-k8s) provides the persistent metadata store that Airflow uses to track DAGs, task instances, and run history.

  • Components layer. A set of purpose-built charms that run the Airflow services and coordinate their configuration. These are described in the Components section below.

Components

The components layer is split into three groups: the core charms that run the Airflow services, the support charms that wire the deployment together, and the external integrators that provide optional data sources.

For the full configuration, actions, and relations of each charm, see the charm reference.

Core charms

The core charms each run one of the long-lived Airflow services inside a workload container. They receive their Airflow configuration and secrets from the coordinator and share the same PostgreSQL metadata database.

  • Airflow API Server (airflow-api-server-k8s) – serves the Airflow web UI and exposes the REST API that clients use to interact with DAGs, runs, and task metadata.

  • Airflow Scheduler (airflow-scheduler-k8s) – monitors DAGs and task instances, resolves their dependencies, and triggers task execution.

  • Airflow DAG Processor (airflow-dag-processor-k8s) – periodically scans the configured DAG folder and parses Python files to discover and update DAG definitions.

  • Airflow Triggerer (airflow-triggerer-k8s) – runs the asynchronous event loop that backs deferrable operators, freeing up worker capacity while tasks wait on external conditions.

Support charms

  • Airflow Coordinator (airflow-coordinator-k8s) is the central support charm. It gathers configuration from every source in the deployment — user configuration, defaults, database credentials, executor settings, DAG source integrators, and provider configuration — merges it into a single consistent Airflow configuration, and shares it with each core charm. As a result, the core charms never need to consume those relations directly.

  • Airflow Provider Configurator (airflow-provider-configurator) lets operators configure Airflow providers as data rather than through charm configuration options. It reads non-sensitive provider configuration from an .ini file in a Git repository and sensitive values from a Juju user secret, then passes the combined result to the coordinator for merging. This keeps the coordinator’s own configuration surface from having to grow with every provider. See Integrate with the Airflow Provider Configurator.

  • Airflow Executor charms configure how Airflow schedules and runs DAG tasks. Each executor charm advertises an executor-specific configuration (such as a Pod template) to the coordinator, which forwards it to the core charms. A deployment can use different executor charms depending on the isolation, scaling, and resource requirements of its workloads. For example, the Airflow Kubernetes Executor runs each task as an isolated Kubernetes Pod.

External integrators

External integrators are optional charms that let Charmed Airflow fetch DAGs and provider configuration from sources outside the Kubernetes cluster. They allow operators to manage DAG distribution and provider configuration without baking either into container images.

  • Git integrator (git-integrator) – provides the connection details and credentials for a remote Git repository containing DAGs or provider configuration. A deployment can run one integrator per repository, so DAGs and provider configuration can live in separate repositories.

  • S3 integrator (s3-integrator) – provides the connection details and credentials for an S3-compatible object storage bucket containing DAGs.

  • Ory Hydra (hydra) – Provides the OAuth configurations required to enable IAM integration.

Integrations

Charmed Airflow components communicate through Juju integrations. The diagram below shows the integrations that connect the components of a Charmed Airflow deployment.

Integrations between the Charmed Airflow components.

Required integrations

The following integrations must be in place for Charmed Airflow to reach an active state:

  • airflow-coordinator – connects each core charm (API server, scheduler, DAG processor, triggerer) to the coordinator. The coordinator sends the merged Airflow configuration to the core charms over this interface.

  • airflow-api-server – used by the coordinator to collect connection information about the API server so it can be embedded in the shared configuration.

  • postgresql_client – used by the coordinator to obtain credentials for the PostgreSQL metadata database, which are then propagated to the core charms.

Optional integrations

The following integrations enable optional capabilities.

  • airflow-kubernetes-executor-config – used by the coordinator to collect the configuration required by the Kubernetes executor, including the Pod template applied when tasks are scheduled as Kubernetes Pods.

  • airflow-provider-configuration – used by the coordinator to collect provider configuration from the Provider Configurator, which it merges into the shared Airflow configuration.

  • git – used by the coordinator to collect the connection information required to fetch DAGs from a Git repository, and by the Provider Configurator to fetch its provider configuration file, through the Git integrator.

  • ingress – used by the API server to obtain a proxied, externally reachable endpoint from traefik-k8s.

  • s3 – used by the coordinator to collect the connection information required to fetch DAGs from an S3-compatible bucket, through the S3 integrator.

  • oauth – Used by the coordinator to retrieve OAuth configurations from the Canonical Identity Platform to enable IAM integration.