[BEAM-5053] CreateRequestsPerMinutePerUser quota limit exceeded

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: test-failures, testing
• Assignee: Mikhail Gryzykhin
• Reporter: Mikhail Gryzykhin
• Created: 31/Jul/18 23:22
• Updated: 16/May/20 13:22
• Resolved: 14/Aug/18 16:09

Description

https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/553/consoleFull

 

Dataflow has a CreateRequestsPerMinutePerUser quota that is 60 requests per minute.

Our Java Validate Runner tests spin up more than 250 jobs in a run.

We should look into best way to fix the issue. Currently, I see following options:

1. Add retry logic. Although this limits us to 1 dataflow job start per second for whole Jenkins. In long scale this can also block one test job if other jobs take all the slots.
2. Utilize different users to spin Dataflow jobs.
3. Find way to rise quota limit on Dataflow. By default the field limits value to 60 requests per minute.

Issue Links

• links to: GitHub Pull Request #6125