[FLINK-27075] Migrate CI from Azure to Github Actions

• Type: Technical Debt
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Build System / CI
• Labels: github-actions
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 06/Apr/22 06:49
• Updated: 20/Dec/23 11:55

Description

Current base commit: master@011b6b44 (related AzureCI)

Issue Links

• is blocked by: FLINK-27069 Fix the potential memory corruption in Thread Mode
• is related to: FLINK-26658 Migrate documentation build to Github Actions
• is related to: FLINK-27488 Migrate flink-docker CI to github actions
• is superceded by: FLINK-33901 Trial Period: GitHub Actions
• Testing discovered: FLINK-33578 Remove scala-2.12 system variable
• links to: Branch for initial POC in XComp/flink fork

Sub-Tasks

• Setup runners: Open
• Setup per-commit builds: Open
• Setup cron jobs: Open
• Enable builds for pull requests: Open
• Remove Azure integration: Open
• misc module: YARN tests are flaky: Closed
• .scalafmt.conf cannot be found in Test packaging/licensing job: Resolved
• finegrained_resource_management module failed: Closed
• IllegalArgumentException in NoticeFileChecker: Resolved
• LicenseChecker fails in GHA but succeeds in Azure: Resolved
• test_pyflink.sh fails: zip is missing in Docker image: Resolved
• Improve speed of compile step: Resolved
• Fatal error due to dubious ownership in docs-404-check: Resolved
• Remove Kafka/Gelly module: Closed
• Think about installing Java instead of providing it through Docker: Open
• Upgrading to actions/checkout@v4 requires GLIBC 2.25, 2.27, or 2.28 to be installed, apparently: Closed
• tests stage: HybridShuffleITCase.testHybridSelectiveExchangesRestart due to timeout: Closed
• e2e 2 stage: PyFlink end-to-end test: Closed
• core stage: 137 exit code: Resolved
• core stage: WebFrontendBootstrapTest.testHandlersMustBeLoaded: Closed
• core stage: HistoryServerStaticFileServerHandlerTest.testRespondWithFile(Path): Closed
• e2e 1 stage: Wordcount on Docker test (custom fs plugin): Resolved
• HAJobRunOnHadoopS3FileSystemITCase failed due to NoSuchMethodError: Resolved
• FineGrainedSlotManagerTest failed with fatal error: Closed
• SqlGatewayE2ECase failed due to ConnectException: Resolved
• Build artifacts cannot be extracted smoothly: Closed
• HiveITCase.testHiveDialect and HiveITCase.testReadWriteHive are failing: Closed
• JoinITCase.testRightOuterJoin failed due to heartbeat timeout: Open
• Run Kubernetes IT test fails due to wrong Maven version: Closed
• Supported maximum parallel jobs: Open
• Test CI on arm: Open
• "DataSet allround end-to-end test" failed due to certificate error: Resolved
• Job (and its data) is lost: Open
• FileUtilsTest.testDeleteDirectoryConcurrently failed (no progress/output): Open
• SemiJoinITCase.testRewriteScalarQueryWithCorrelation3 failed due to heartbeat timeout: Open
• composite actions do not work in release branchs where GHA work wasn't backported: Open
• Add GHA support in release-1.17 branch: Closed
• Docker pull failed: Open
• Hosted runner lost communication: Closed
• Python e2e test fails with exit code 1: Open
• StreamTaskSelectiveReadingTest.testReadFinishedInput(StreamTaskSelectiveReadingTest timed out: Open
• ActiveResourceManagerFactoryTest.testGetEffectiveConfigurationForResourceManagerCoarseGrained failed with 239 exit code: Open
• SourceStreamTaskTest.testTriggeringStopWithSavepointWithDrain failed due async checkpoint triggering not being completed: Closed
• Various HiveDialectQueryITCase tests fail in GitHub Actions workflow with Hadoop 3.1.3 enabled: Resolved
• ScalaSerializersMigrationTest.testStableAnonymousClassnameGeneration fails in GHA with JDK 17 and 21: Resolved
• RetryingRegistrationTest.testRetryConnectOnFailure failed: Open