Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39515

Improve/recover scheduled jobs in GitHub Actions

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Blocker
    • Resolution: Done
    • 3.4.0
    • 3.4.0
    • Build, Project Infra
    • None

    Description

      There are five problems to address.

      First, the scheduled jobs are broken as below:

      https://github.com/apache/spark/actions/runs/2513261706
      https://github.com/apache/spark/actions/runs/2512750310
      https://github.com/apache/spark/actions/runs/2509238648
      https://github.com/apache/spark/actions/runs/2508246903
      https://github.com/apache/spark/actions/runs/2507327914
      https://github.com/apache/spark/actions/runs/2506654808
      https://github.com/apache/spark/actions/runs/2506143939
      https://github.com/apache/spark/actions/runs/2502449498
      https://github.com/apache/spark/actions/runs/2501400490
      https://github.com/apache/spark/actions/runs/2500407628
      https://github.com/apache/spark/actions/runs/2499722093
      https://github.com/apache/spark/actions/runs/2499196539
      https://github.com/apache/spark/actions/runs/2496544415
      https://github.com/apache/spark/actions/runs/2495444227
      https://github.com/apache/spark/actions/runs/2493402272
      https://github.com/apache/spark/actions/runs/2492759618
      https://github.com/apache/spark/actions/runs/2492227816

      See also https://github.com/apache/spark/pull/36899 or https://github.com/apache/spark/pull/36890
      In the master branch, seems like at least Hadoop 2 build is broken currently.

      Second, it is very difficult to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one.

      Since GitHub added the feature to import other workflow, we should leverage this feature, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test_ansi.yml and https://docs.github.com/en/actions/using-workflows/reusing-workflows. Once we can separate them, it will be defined as a separate workflow.

      Namely, each scheduled job should be classified under "All workflows" at https://github.com/apache/spark/actions so other developers can easily track them.

      Third, we should set the scheduled jobs for branch-3.3, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L78-L83 for branch-3.2 job.

      Forth, we should improve duplicated test skipping logic. See also https://github.com/apache/spark/pull/36413#issuecomment-1157205469 and https://github.com/apache/spark/pull/36888

      Fifth, we should probably replace the base image (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain ubunto image w/ Docker image cache. See also https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md

      Attachments

        Issue Links

          1.
          Set a scheduled build for branch-3.3 Sub-task Resolved Hyukjin Kwon
          2.
          Recover branch-3.2 build broken by is-changed.py script missing Sub-task Resolved Hyukjin Kwon
          3.
          Define each workflow for each scheduled job in GitHub Actions Sub-task Resolved Hyukjin Kwon
          4.
          Refactor and merge all related job selection logic into precondition Sub-task Resolved Hyukjin Kwon
          5.
          Add Apache Spark infra GA image cache Sub-task Resolved Yikun Jiang
          6.
          Move checkout and sync steps into re-usable composite action Sub-task Resolved Unassigned
          7.
          Downgrade scala-maven-plugin to 4.6.1 Sub-task Resolved Yang Jie
          8.
          Test failure in SPARK-39387 with JDK 11 Sub-task Resolved Yang Jie
          9.
          ExpressionSetSuite test failure with Scala 2.13 Sub-task Resolved Yang Jie
          10.
          JDBCV2Suite fails with Scala 2.13 Sub-task Resolved Bruce Robbins
          11.
          GeneratorFunctionSuite fails with Scala 2.13 Sub-task Resolved Bruce Robbins
          12.
          AdaptiveQueryExecSuite fails with Scala 2.13 Sub-task Resolved Bruce Robbins
          13.
          Plan stability test is broken with Scala 2.13 Sub-task Resolved Bruce Robbins
          14.
          Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING Sub-task Resolved Yang Jie
          15.
          Add safe.directory for container based job Sub-task Resolved Yikun Jiang
          16.
          PySpark support numpy 1.23.X Sub-task Resolved Yikun Jiang
          17.
          Disable scheduled builds that do not pass even once Sub-task Resolved Hyukjin Kwon
          18.
          Add REFRESH_DATE flag and use previous cache to build cache image Sub-task Resolved Yikun Jiang
          19.
          Enable base image build in PySpark job Sub-task Resolved Yikun Jiang
          20.
          Enable base image build in lint job and fix sparkr env Sub-task Resolved Yikun Jiang
          21.
          Enable base image build in SparkR job Sub-task Resolved Yikun Jiang
          22.
          Support numpy 1.23.0 (Remove numpy<1.23.0 version limit) Sub-task Resolved Yikun Jiang

          Activity

            People

              yikunkero Yikun Jiang
              hyukjin.kwon Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: