Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-20978

Hitting cross-project concurrency limit in GitHub Actions

    XMLWordPrintableJSON

    Details

    • Type: Wish
    • Status: Closed
    • Priority: Major
    • Resolution: Incomplete
    • Fix Version/s: None
    • Component/s: Github
    • Labels:
      None
    • Project:
      Airflow

      Description

      We are starting to hit - more and more often - concurrency limits in GitHub Actions where we seem to compete with other Apache projects.

      From the GitHub support ticket it seems that yesterday we had some real and huge delays in our builds (> 3 hours) because of this very problem. We have - apparently 180 concurrency limit for all Apache projects.

      https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#usage-limits

      We recently moved to much more optimized builds which are far smaller and we run much lower number of those.

      You can see PRS which I merged recently:
      * https://github.com/apache/airflow/pull/11402
      * https://github.com/apache/airflow/pull/11417
      * https://github.com/apache/airflow/pull/11471
      * https://github.com/apache/airflow/pull/11460



      We've also implemented automated cancelling of duplicate workflows (including a custom cancel-workflow-run Action I developed: https://github.com/potiuk/cancel-workflow-runs and shared with Skywalking and Apache Beam projects.

      But I think we are getting close to the limits of what we can do to minimise the strain on the jobs from our project, also we have no influence (besides Skywalking and Beam) on other projects.

      Is there something we can do here? Guaranteed number of runners per project? Negotiating with GitHub higher concurrency ?

      It will become a similar problems as we had with Travis, if we do not take any coordinated action here.

      I will continue working on optimising our builds (there are still some small things I can optimise),

      I am also happy to donate my time to help other apache projects to learn from that and optimise the use of Github Actions, but I do not have enough information that I could act on - like to know which projects are using resources etc.

      Happy to be involved in solving the problem if I get some help from infra on those!

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              potiuk Jarek Potiuk
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: