[FLINK-34989] Apache Infra requests to reduce the runner usage for a project

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.19.0, 1.18.1, 1.20.0
• Component/s: Build System / CI
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 02/Apr/24 11:35
• Updated: 05/Jun/24 17:38

Description

The GitHub Actions CI utilizes runners that are hosted by Apache Infra right now. These runners are limited. The runner usage can be monitored via the following links:

• Flink-specific report (needs ASF committer rights) This project-specific report can only be modified through the HTTP GET parameters of the URL.
• Global report (needs ASF membership)

There was a policy change announced recently:

Policy change on use of GitHub Actions

Due to misconfigurations in their builds, some projects have been using
unsupportable numbers of GitHub Actions. As part of fixing this situation,
Infra has added a 'resource use' section to the policy on GitHub Actions.
This section of the policy will come into effect on April 20, 2024:

All workflows MUST have a job concurrency level less than or equal to 20.
This means a workflow cannot have more than 20 jobs running at the same time
across all matrices.
All workflows SHOULD have a job concurrency level less than or equal to 15.
Just because 20 is the max, doesn't mean you should strive for 20.
The average number of minutes a project uses per calendar week MUST NOT
exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200
hours).
The average number of minutes a project uses in any consecutive five-day
period MUST NOT exceed the equivalent of 30 full-time runners (216,000
minutes, or 3,600 hours).
Projects whose builds consistently cross the maximum use limits will lose
their access to GitHub Actions until they fix their build configurations.
The full policy is at
https://infra.apache.org/github-actions-policy.html.

Currently (last week of March 2024) Flink was ranked at #19 of projects that used the Apache Infra runner resources the most which doesn't seem too bad. This contained not only Apache Flink but also the Kubernetes operator, connectors and other resources. According to this source Apache Infra manages 180 runners right now.

Issue Links

• is related to: FLINK-34331 Enable Apache INFRA ephemeral runners for nightly builds
• relates to: FLINK-34937 Apache Infra GHA policy update
• links to: GitHub Pull Request #24606