[SPARK-48094] Reduce GitHub Action usage according to ASF project allowance

• Type: Umbrella
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Project Infra
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 02/May/24 20:14
• Updated: 10/May/24 05:40

Description

ASF INFRA POLICY

• https://infra.apache.org/github-actions-policy.html

MONITORING

• https://infra-reports.apache.org/#ghactions&project=spark&hours=168

TARGET

• All workflows MUST have a job concurrency level less than or equal to 20. This means a workflow cannot have more than 20 jobs running at the same time across all matrices.
• All workflows SHOULD have a job concurrency level less than or equal to 15. Just because 20 is the max, doesn't mean you should strive for 20.
• The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).
• The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours).

DEADLINE

17th of May, 2024

Issue Links

• is related to: SPARK-44111 Prepare Apache Spark 4.0.0

Sub-Tasks

• Run `build_non_ansi.yml` once per day: Resolved
• Run `build_maven_java21_macos14.yml` every two days: Resolved
• Limit GHA job execution time to up to 3 hours in `build_and_test.yml`: Resolved
• Enable `NOLINT_ON_COMPILE` for all except `lint` job: Resolved
• Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`: Resolved
• Run `publish_snapshot.yml` once per day: Resolved
• Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`: Resolved
• Skip `tpcds-1g` and `docker-integration-tests` tests from `RocksDB UI-Backend` job: Resolved
• Enable `k8s-integration-tests` only for `kubernetes` module change: Resolved
• Remove all Maven compilation build: Resolved
• Move tpcds-1g and docker-integration-tests to daily scheduled jobs: Resolved
• Remove `Python 3.11` from `build_python.yml`: Resolved
• Run `pyspark-pandas*` only in PR builder and Daily Python CIs: Resolved
• Run `k8s-integration-tests` only in PR builder and Daily CIs: Resolved
• Run `sparkr` only in PR builders and Daily CIs: Resolved
• Run `buf` and `ui` only in PR builders and Java 21 Daily CI: Resolved
• Run `yarn` test only in PR builders and Maven Daily CIs: Resolved
• Serialize `build_python.yml` to run a single Python version per cron schedule: Closed
• Run `build` job of `build_and_test.yml` only if needed: Resolved
• Merge `connect` back to the original test pipeline: Resolved
• Run `build/scala-213/java-11-17` jobs of branch-3.5 only if needed: Resolved
• Run `docs` only in PR builders and `build_non_ansi` Daily CI: Resolved
• Enable TPC-DS and docker tests in forked repository: Resolved
• Make `maven-deploy-plugin` retry 3 times: Resolved
• Split `build_python.yml` into per-version cron jobs: Resolved
• Spin off `pyspark` tests from `build_branch35.yml` Daily CI: Resolved
• Spin off `pyspark` tests from `build_branch34.yml` Daily CI: Resolved
• Run `build/scala-213/java-11-17` jobs of `branch-3.4` only if needed: Resolved