[BEAM-14263] beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing consistently

• Type: Bug
• Status: Open
• Priority: P1
• Resolution: Unresolved
• Component/s: test-failures
• Labels: currently-failing stale-assigned
• Assignee: Reuven Lax
• Reporter: Daniel Oliveira
• Created: 05/Apr/22 22:12
• Updated: 05/Jun/22 01:22

Description

testBigQueryStorageWrite30MProto seems to have been failing since being originally introduced. First failure I found: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2/1391/, which includes when this PR was merged: https://github.com/apache/beam/pull/17038

I don't see any explicit reason this might be failing either in the console logs or Dataflow logs. Here's the console logs:

java.lang.RuntimeException: Workflow failed. Causes: S06:WriteToBQ/StorageApiLoads/GroupIntoBatches/ParDo(GroupIntoBatches)/ParMultiDo(GroupIntoBatches)/Reshard/Read+WriteToBQ/StorageApiLoads/GroupIntoBatches/ParDo(GroupIntoBatches)/ParMultiDo(GroupIntoBatches)/Reshard/UnreifyWindow+WriteToBQ/StorageApiLoads/GroupIntoBatches/ParDo(GroupIntoBatches)/ParMultiDo(GroupIntoBatches)+WriteToBQ/StorageApiLoads/StorageApiWriteSharded/Write Records/ParMultiDo(WriteRecords)/Reshard/Reify+WriteToBQ/StorageApiLoads/StorageApiWriteSharded/Write Records/ParMultiDo(WriteRecords)/Reshard/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: 

  testpipeline-jenkins-0319-03190548-sfjp-harness-263n
      Root cause: The worker lost contact with the service.,

Issue Links

• is a child of: BEAM-14276 beam_PostCommit_Java_DataflowV2 failures parent bug
• links to: GitHub Pull Request #17325