[BEAM-6528] Dataflow jobs failing in multiple PostCommits, including Java PostCommit.

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: runner-dataflow, test-failures
• Labels: currently-failing flake
• Assignee: Daniel Oliveira
• Reporter: Daniel Oliveira
• Created: 28/Jan/19 22:56
• Updated: 16/May/20 13:29
• Resolved: 29/Jan/19 18:46

Description

• Jenkins Job
• Gradle Build Scan

Initial investigation:

It seems this test has failed four times just today with flakes in various integration tests. I've linked to the first instance of this failure above.

In the Dataflow dashboard I see this error:

Startup of the worker pool in zone us-central1-f failed to bring up any of the desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone 'projects/apache-beam-testing/zones/us-central1-f' does not have enough resources available to fulfill the request. '(resource type:pd-standard)'.

Possibly some quota issue?

 

Edit:

I'm updating this bug because it seems like this isn't restricted to beam_PostCommit_Python_Verify after all. I've now seen this happening in various java postcommits:

beam_PostCommit_Java
beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow
beam_PostCommit_Java_ValidatesRunner_Dataflow

In each case the error looks something like this:

java.lang.RuntimeException: Startup of the worker pool in zone us-central1-f failed to bring up any of the desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone 'projects/apache-beam-testing/zones/us-central1-f' does not have enough resources available to fulfill the request.  '(resource type:pd-standard)'.Workflow failed.