Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-5633

Apache Airflow Integration with HashiCorp Nomad

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.10.5
    • Fix Version/s: None
    • Component/s: executors, operators
    • Labels:
      None

      Description

      Nomad is an orchestrator, similar to Kubernetes and used in production by companies like PagerDuty, Target, Citadel, Trivago, Pandora, Roblox, etc.  Built by HashiCorp (company behind other tools such as Terraform, Vault, and Consul).

      We've gotten many requests from our customers and open source community for an integration with Airflow.  There are a variety of community PRs from supportive members who are trying to build their own (linked notable ones below) - our approach would be to build one officially supported integration ourselves for all our users and customers.  

      https://github.com/apache/airflow/pull/2708

      https://github.com/hashicorp/nomad/issues/419

      We'd love to understand how much interest and support we can gauge from the Airflow committee around an integration like this.  Ideally, we'd aim to avoid a situation where our integration is not accepted + Nomad users using Airflow are subject to an incomplete/less-than-ideal experience on a lagging HashiCorp-maintained-fork. 

      This has unfortunately happened with our Spark integration, so we'd love to gauge the Airflow committee early-on in the process before we potentially build the integration.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              yishan Yishan
            • Votes:
              23 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated: