[HDDS-6496] Improve CI checks

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: build, CI
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 23/Mar/22 14:12
• Updated: 17/Apr/24 05:19

Description

Some ideas for improvement:

• compile (11) is not necessary for acceptance test changes, when compilation is necessary only to get the binary tarball for running the tests.
• basic checks are deterministic (except basic (unit) to some degree). Failures can only be fixed by code change, so we can restore fail-fast behavior.
• basic checks are also quick, so we can postpone starting integration until these finish.

Issue Links

• contains: HDDS-5730 Decrease run time of acceptance tests
• is a parent of: HDDS-9000 Speed up unit/integration tests
• relates to: HDDS-5626 Track and Address Flaky tests
• relates to: HDDS-8719 Improve thread names
• relates to: HDDS-6729 Migrate tests to JUnit5

Sub-Tasks

• Skip compile(11) for non-java changes: Resolved
• Reduce OzoneFS test combinations in ozonesecure: Resolved
• Create compat acceptance split: Resolved
• Avoid repeating some S3 tests for different bucket types: Resolved
• Further splits integration tests: Resolved
• Verify build from source release tarball in CI: Resolved
• Omit download progress in builds: Resolved
• Add CI to check pull request title: Resolved
• Increase timeout for basic check: Resolved
• Disallow ellipsis in PR title: Resolved
• Ignore pr_title_check for selective checks: Resolved
• Skip installing awscli if already present: Resolved
• Timed out tests missing from bundle: Resolved
• Separate scheduled CI from push/PR workflows: Resolved
• Suppress warning about long filenames in tar: Resolved
• Kubernetes check should fail fast if cluster cannot start: Resolved
• Acceptance tests need rebalancing: Resolved
• Reorder OM nodes in HA acceptance tests: Resolved
• Reduce S3 acceptance test setup time: Resolved
• Run HA secure tests from s3g container: Resolved
• Disable JaCoCo for PRs and in forks: Resolved
• Selective checks: handle change in junit.sh: Resolved
• Selective checks: handle change in ci.yaml: Resolved
• Increase timeout of CI build: Resolved
• Fail integration tests if RocksDB objects are leaked: Resolved
• Support multiple acceptance test scripts with custom config: Resolved
• Update compose README: Resolved
• Let run.sh start more than 3 datanodes: Resolved
• Selective checks: skip acceptance tests for compose doc changes: Resolved
• Test Hadoop compatibility in Ozone HA environment: Resolved
• Convert s3-haproxy to an add-on: Resolved
• Reduce ozone sh calls in robot tests: Resolved
• Reuse same Ozone cluster for Hadoop tests: Resolved
• Create separate check for native build: Resolved
• Ensure docker env is stopped: Resolved
• error validating kustomization.yaml: Resolved
• Keep separate Robot XMLs if rebot fails to combine them: Resolved
• Rebalance integration tests: Resolved
• Update instructions in dependency check: Resolved
• Add dedicated log file for each service in docker test: Resolved
• Error creating stack dump in S3 HA environment: Resolved
• Skip acceptance/kubernetes for license changes: Resolved
• Extract nodes required for SCM decommission test: Resolved
• Create separate acceptance split for cert rotation: Resolved
• Create a script to list all acceptance test splits: Resolved
• Merge unit and integration checks into a combined one: Resolved
• Require successful basic checks for long-running tests: Resolved
• Summarize failures of acceptance/kubernetes checks: Resolved
• Mark disabled tests using custom tag/category: Resolved
• Intermittent error while removing docker network: Resolved
• Handle empty unit check: Resolved
• Fail checks in Summary step instead of Test: Resolved
• Introduce a new workflow for testing intermittently failing unit tests.: Resolved
• Parameter for fail-fast behavior in flaky-test-check: Resolved
• Upload artifacts of unit/native check: Resolved
• Allow disabling JUnit timeouts: Resolved
• Increase timeout of unit check: Resolved
• Group output of check steps: Resolved
• Improve flaky-test-check job name.: Resolved
• Remove scheduled CI run: Resolved
• Skip unhealthy tests in repeated runs workflow: Resolved
• Redirect pre-test build output for integration check: Resolved
• Introduce FlakyTest category for flaky JUnit4 tests: Resolved
• Rebalance acceptance tests: Resolved
• Count total number of failures in flaky-test-check: Resolved
• Do not run all checks for draft PR touching CI files: Resolved
• Resolve dependabot build issues when updating npm packages: Resolved
• Create workflow for running acceptance test repeatedly: Resolved
• Skip coverage check for dependabot: Resolved
• Skip push build for dependabot: Resolved
• Use secure cluster for upgrade acceptance tests: Resolved
• Check that dependencies have acceptable license: Resolved
• Set descriptive title for upgrade Robot tests: Resolved
• Create stack dump if cannot remove docker network: Resolved
• Checkstyle check passing despite config error: Resolved
• Selective checks: skip tests for IntelliJ config change: Resolved
• Split parts of misc suite: Resolved
• Intermittent failure at hadoop-maven-plugin in compile check: Resolved
• Simplify Maven cache for dependencies in CI: Resolved
• Let junit.sh fail if no tests were matched in repeated run: Resolved
• Pre-build Ozone in flaky-test-check: Resolved
• Collect logs from cancelled integration test: Resolved
• Enable Surefire process checker: Resolved
• Remove validation of test method in flaky-test-check: Resolved
• Node.js 16 actions are deprecated: Resolved
• Increase timeout for compile check: Resolved
• Group dependabot PRs for frontend dependencies: Resolved
• Download zlib fails with 403 Forbidden in CI: Resolved
• Intermittent failure downloading from sourceware.org: Resolved
• Add check for dependencies included in Ozone FS shaded jar: Open
• No logs if pod stuck in PodInitializing: Open
• Consistently enable exit-on-error in test scripts: Open
• Allow running flaky-test-check with specific Ratis commit: Resolved
• Allow running nested test classes in flaky-test-check: Resolved
• Intermittent build failure while downloading nodejs: Resolved
• Selective checks: skip tests for Dashboard changes: Resolved
• Schedule dependabot for weekend: Resolved
• Populate dependency cache in separate workflow: Resolved
• Fetch dependencies using actual build: Resolved
• Repeated tests stop if fork does not return: Resolved
• Downgrade Surefire to 3.0.0-M4: Resolved
• Checkstyle summary excludes errors with "xml": Resolved
• Exclude proto3 classes from coverage: Resolved
• Rename unit check to native: Resolved
• Remove custom dependabot workflow: Resolved
• Only bump npm packages for security: Resolved
• Selective checks: trigger native check in more cases: Resolved
• Integration check no longer needs Ozone repo: Resolved
• Run full Ozone CI with specific Ratis commit: Resolved
• flaky-test-check does not list failed iterations: Resolved
• Enable Gradle remote cache in CI: Open
• Migrate from Gradle Enterprise to Develocity: Resolved
• Avoid publishing build scan for helper Maven commands: Resolved
• flaky-test-check should retain of the failed logs: Resolved
• Selective checks: integration skipped when build not required: Resolved
• CI with Ratis ignores Ozone ref: Resolved
• Verify javadoc creation in CI: Resolved
• Add smoketest in ozone-runner image CI: Open
• Require successful dependency/licence checks for acceptance/compile/kubernetes: Resolved
• Selective checks: trigger checkstyle for properties file changes: Resolved
• Add check for javadoc correctness: Resolved
• Avoid unnecessary duplicate build: Resolved
• Bump macOS runner version to macos-13: Resolved
• Let coverage report link to java sources: Resolved
• Make MAVEN_OPTS optional: Resolved
• Provide details about integration check failure: Resolved
• Use ozone-runner from GitHub in CI: Resolved
• Compile failure is ignored in build check: Resolved
• Hadoop download failure not reported as error: Resolved
• Skip shade in most integration checks: Resolved
• Use ozone-testkrb5 from GitHub: Resolved
• Use ozone image from GitHub in CI: Resolved
• Use ozone-runner from GitHub in ozone image build: Resolved
• Extract scheduled workflow for populate-cache: Resolved
• Skip spotbugs for modules with only generated code: Resolved
• Use hadoop images from GitHub in CI: Resolved
• Some CI checks passing despite error: Patch Available
• Use Java 21 in CI: Patch Available
• Run some acceptance tests on Mac: Resolved
• Require successful quick checks for repro: Patch Available
• Logs missing if kubernetes check fails before tests: Open