Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-2499

Dockerised CI pipeline

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.10.0, 2.0.0
    • Fix Version/s: 1.10.2
    • Component/s: ci, tests
    • Labels:

      Description

      PR: https://github.com/apache/incubator-airflow/pull/3393

      Currently, running unit tests is a difficult process. Airflow tests depend on many external services and other custom setup, which makes it hard for contributors to work on this codebase. CI builds have also been unreliable[0][1][2][3], and it is hard to reproduce the causes. Having contributors trying to emulate the build environment every time makes it easier to get to an "it works on my machine" sort of situation.

      This PR implements a dockerised version of the current build pipeline. This setup has a few advantages:

      • TravisCI tests are reproducible locally
      • The same build setup can be used to create a local development environment (there's a request for it [4])

       

      Implementation details

      • I'm using Docker Compose for the container orchestration and configuration.
      • MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running inside their own containers
      • I created a separate repo, called incubator-airflow-ci[5] (TravisCI build here[6]), where a base image with all dependencies is built. In this case, I'm following the same pattern the CouchDB[7] project follows
      • Hadoop, Hive and MiniCluster were moved to this base image
      • The current TravisCI pipeline lives here[8]. A few tests are still failing. It's still WIP.

       

      References

      [0] https://issues.apache.org/jira/browse/AIRFLOW-671
      [1] https://issues.apache.org/jira/browse/AIRFLOW-968
      [2] https://issues.apache.org/jira/browse/AIRFLOW-2157
      [3] https://issues.apache.org/jira/browse/AIRFLOW-2272
      [4] https://issues.apache.org/jira/browse/AIRFLOW-1042
      [5] https://github.com/gerardo/incubator-airflow-ci
      [6] https://travis-ci.org/gerardo/incubator-airflow-ci
      [7] https://travis-ci.org/apache/couchdb-ci
      [8] https://travis-ci.org/gerardo/incubator-airflow

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gcuriel Gerardo Curiel
                Reporter:
                gcuriel Gerardo Curiel
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 72h
                  72h
                  Remaining:
                  Remaining Estimate - 72h
                  72h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified