[SPARK-32244] Build and run the Spark with test cases in Github Actions

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 2.4.6, 3.0.0, 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Project Infra
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 09/Jul/20 13:35
• Updated: 12/Dec/22 18:10
• Resolved: 04/Jan/21 05:58

Description

Last week and onwards, the Jenkins machines became very unstable for some reasons.

• Apparently, the machines became extremely slow. Almost all tests can't pass.
• One machine (worker 4) started to have the corrupt .m2 which fails the build.
• Documentation build fails time to time for an unknown reason in Jenkins machine specifically.

Almost all PRs are basically blocked by this instability currently.

This JIRA aims to run the tests in Github Actions.

• To avoid depending on few persons who can access to the cluster.
• To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce.
• To control the environment more flexibly.
• Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost.

Issue Links

• contains: SPARK-32246 Enable streaming-kinesis-asl tests in Github Action CI

Sub-Tasks

• Implement the base to run Spark tests in GitHun Actions: Resolved
• scipy installation fails with PyPy: Resolved
• Recover JDK 11 builds in Github Actions: Resolved
• Run Github Actions builds in other branches as well: Resolved
• Reenable MasterSuite's "Master should avoid dead loop while launching executor failed in Worker": Resolved
• Enable doctests in run-tests.py back: Resolved
• Make readability better in the test result logs: Resolved
• Reenable SparkSQLEnvSuite's "external listeners should be initialized with Spark classloader": Closed
• Set jobs that runs periodically for other profile combinations: Resolved
• More resources in Github Actions: Resolved
• Run smoke tests after a commit is pushed: Resolved
• Run only relevant builds in parallel at Github Actions: Resolved
• Flaky Test: submit a barrier ResultStage that requires more slots than current total under local-cluster mode: Resolved
• Flaky Test: YarnClusterSuite 4 test cases: Resolved
• Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions: Resolved
• Flaky Test: ExecutorAllocationManagerSuite.add executors default profile: Resolved
• Test PySpark with Python 3.8 in Github Actions as well: Resolved
• Publish failed and succeeded test reports in GitHub Actions: Resolved
• Enable crossPaths back to prevent side effects: Resolved
• Leverage Conda environment at pip packaging test in GitHub Actions: Resolved
• Don't skip pandas UDF tests in IntegratedUDFTestUtils: Resolved
• Do not install SparkR in test-only mode in testing script: Resolved
• Manually install R instead of using setup-r in GitHub Actions: Resolved
• Include GitHub Action file as the changes in testing: Resolved
• Installs qpdf package for CRAN check in GitHub Actions: Resolved
• Remove the fork of action-surefire-report in test_report.yml: Resolved
• Remove the fork of action-download-artifact in test_report.yml: Resolved
• Upload unit-tests.log as an artifact: Resolved
• Use workflow_dispatch to enable manual test triggers: Resolved
• Add 'build' and 'project/build.properties' into cache key of SBT and Zinc: Resolved
• Skip test result report if no JUnit XML files are found: Resolved
• Enables 'fail_if_no_tests' when reporting test results: Resolved
• Add GitHub Action job for Hadoop 2: Resolved
• Add/remove (un)necessary cache and restructure GitHub Actions yaml: Resolved