Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-20978

Hitting cross-project concurrency limit in GitHub Actions



    • Type: Wish
    • Status: Closed
    • Priority: Major
    • Resolution: Incomplete
    • Fix Version/s: None
    • Component/s: Github
    • Labels:
    • Project:


      We are starting to hit - more and more often - concurrency limits in GitHub Actions where we seem to compete with other Apache projects.

      From the GitHub support ticket it seems that yesterday we had some real and huge delays in our builds (> 3 hours) because of this very problem. We have - apparently 180 concurrency limit for all Apache projects.


      We recently moved to much more optimized builds which are far smaller and we run much lower number of those.

      You can see PRS which I merged recently:
      * https://github.com/apache/airflow/pull/11402
      * https://github.com/apache/airflow/pull/11417
      * https://github.com/apache/airflow/pull/11471
      * https://github.com/apache/airflow/pull/11460

      We've also implemented automated cancelling of duplicate workflows (including a custom cancel-workflow-run Action I developed: https://github.com/potiuk/cancel-workflow-runs and shared with Skywalking and Apache Beam projects.

      But I think we are getting close to the limits of what we can do to minimise the strain on the jobs from our project, also we have no influence (besides Skywalking and Beam) on other projects.

      Is there something we can do here? Guaranteed number of runners per project? Negotiating with GitHub higher concurrency ?

      It will become a similar problems as we had with Travis, if we do not take any coordinated action here.

      I will continue working on optimising our builds (there are still some small things I can optimise),

      I am also happy to donate my time to help other apache projects to learn from that and optimise the use of Github Actions, but I do not have enough information that I could act on - like to know which projects are using resources etc.

      Happy to be involved in solving the problem if I get some help from infra on those!




            • Assignee:
              potiuk Jarek Potiuk
            • Votes:
              0 Vote for this issue
              5 Start watching this issue


              • Created: