Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32244

Build and run the Spark with test cases in Github Actions

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Critical
    • Resolution: Done
    • 2.4.6, 3.0.0, 3.1.0
    • 3.1.0
    • Project Infra
    • None

    Description

      Last week and onwards, the Jenkins machines became very unstable for some reasons.

      • Apparently, the machines became extremely slow. Almost all tests can't pass.
      • One machine (worker 4) started to have the corrupt .m2 which fails the build.
      • Documentation build fails time to time for an unknown reason in Jenkins machine specifically.

      Almost all PRs are basically blocked by this instability currently.

      This JIRA aims to run the tests in Github Actions.

      • To avoid depending on few persons who can access to the cluster.
      • To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce.
      • To control the environment more flexibly.
      • Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost.

      Attachments

        Issue Links

          contains

          Bug - A problem which impairs or prevents the functions of the product. SPARK-32246 Enable streaming-kinesis-asl tests in Github Action CI

          • Major - Major loss of function.
          • Resolved
          1.
          		 Implement the base to run Spark tests in GitHun Actions Sub-task Resolved Hyukjin Kwon
          2.
          		 scipy installation fails with PyPy Sub-task Resolved Hyukjin Kwon
          3.
          		 Recover JDK 11 builds in Github Actions Sub-task Resolved Dongjoon Hyun
          4.
          		 Run Github Actions builds in other branches as well Sub-task Resolved Hyukjin Kwon
          5.
          		 Reenable MasterSuite's "Master should avoid dead loop while launching executor failed in Worker" Sub-task Resolved wuyi
          6.
          		 Enable doctests in run-tests.py back Sub-task Resolved Hyukjin Kwon
          7.
          		 Make readability better in the test result logs Sub-task Resolved Gengliang Wang
          8.
          		 Reenable SparkSQLEnvSuite's "external listeners should be initialized with Spark classloader" Sub-task Closed Unassigned
          9.
          		 Set jobs that runs periodically for other profile combinations Sub-task Resolved Unassigned
          10.
          		 More resources in Github Actions Sub-task Resolved Holden Karau
          11.
          		 Run smoke tests after a commit is pushed Sub-task Resolved Dongjoon Hyun
          12.
          		 Run only relevant builds in parallel at Github Actions Sub-task Resolved Hyukjin Kwon
          13.
          		 Flaky Test: submit a barrier ResultStage that requires more slots than current total under local-cluster mode Sub-task Resolved Unassigned
          14.
          		 Flaky Test: YarnClusterSuite 4 test cases Sub-task Resolved Unassigned
          15.
          		 Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions Sub-task Resolved Jungtaek Lim
          16.
          		 Flaky Test: ExecutorAllocationManagerSuite.add executors default profile Sub-task Resolved Thomas Graves
          17.
          		 Test PySpark with Python 3.8 in Github Actions as well Sub-task Resolved Hyukjin Kwon
          18.
          		 Publish failed and succeeded test reports in GitHub Actions Sub-task Resolved Hyukjin Kwon
          19.
          		 Enable crossPaths back to prevent side effects Sub-task Resolved Hyukjin Kwon
          20.
          		 Leverage Conda environment at pip packaging test in GitHub Actions Sub-task Resolved Hyukjin Kwon
          21.
          		 Don't skip pandas UDF tests in IntegratedUDFTestUtils Sub-task Resolved Hyukjin Kwon
          22.
          		 Do not install SparkR in test-only mode in testing script Sub-task Resolved Hyukjin Kwon
          23.
          		 Manually install R instead of using setup-r in GitHub Actions Sub-task Resolved Hyukjin Kwon
          24.
          		 Include GitHub Action file as the changes in testing Sub-task Resolved Hyukjin Kwon
          25.
          		 Installs qpdf package for CRAN check in GitHub Actions Sub-task Resolved Hyukjin Kwon
          26.
          		 Remove the fork of action-surefire-report in test_report.yml Sub-task Resolved Hyukjin Kwon
          27.
          		 Remove the fork of action-download-artifact in test_report.yml Sub-task Resolved Hyukjin Kwon
          28.
          		 Upload unit-tests.log as an artifact Sub-task Resolved Hyukjin Kwon
          29.
          		 Use workflow_dispatch to enable manual test triggers Sub-task Resolved Takeshi Yamamuro
          30.
          		 Add 'build' and 'project/build.properties' into cache key of SBT and Zinc Sub-task Resolved Hyukjin Kwon
          31.
          		 Skip test result report if no JUnit XML files are found Sub-task Resolved Unassigned
          32.
          		 Enables 'fail_if_no_tests' when reporting test results Sub-task Resolved Unassigned
          33.
          		 Add GitHub Action job for Hadoop 2 Sub-task Resolved Dongjoon Hyun
          34.
          		 Add/remove (un)necessary cache and restructure GitHub Actions yaml Sub-task Resolved Hyukjin Kwon

          Activity

            People

              gurwls223 Hyukjin Kwon
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: