Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6812

KubernetesJobOperator for airflow

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.10.3, 2.0.0, 1.10.9
    • Fix Version/s: None
    • Component/s: DAG, operators
    • Labels:

      Description

      Dear reader,

      I have created an Airflow Kubernetes Job operator that allows running a Job in kubernetes. I follows similar to the KubernetesPodOperator.

      What are the steps, and what would be required to add this to airflow?

      Improves: Adds an operator for kubernetes Jobs.

      Repohttps://github.com/LamaAni/KubernetesJobOperator

      Why/when (to) use this over the KubernetesPodOperator:
      The kubernetes Job allows for more execution options such as retries/timeouts/deadlines/replicas/etc.. which cannot be defined directly on a pod.

      Also, the connection between the kubernetes pod and the airflow worker can be lost, due to communication issues,
      pod deletions or just pod scheduling problems in the cluster. The Kubernetes Job is a "definition" like resource, and therefore would lose its execution state only if deliberately deleted. A job will also recover automatically from pod manual deletions and pod scheduling errors.

      You can find a description of the kubernetes Job resource [here](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              LamaAni Zav Shotan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: