[BEAM-13931] BigQueryIO is sending rows that are too large to Deadletter Queue even on RETRY_ALWAYS

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Affects Version/s: 2.35.0, 2.36.0
• Fix Version/s: 2.37.0
• Component/s: io-java-gcp
• Assignee: Pablo Estrada
• Reporter: Pablo Estrada
• Created: 11/Feb/22 23:42
• Updated: 15/Feb/22 22:09
• Resolved: 15/Feb/22 22:09

Description

Note that BQ does not support requests over a certain size, and rows that go past the size may be output into a dead-letter queue that they can get back with BigQueryIO.Write.Result.getFailedInsertsWithErr

A change went into Beam that outputs rows into the BQIO DLQ even if they're meant to be retried indefinitely.

https://github.com/apache/beam/commit/1f08d1f3ddc2e7bc7341be4b29bdafaec18de9cc#diff-26dbe8f625f702ae3edacdbc02b12acc6e423542fe16835229e22ef8eb4e109cR979-R989
 
 
A workaround is to set this pipeline option to a larger amount: https://github.com/apache/beam/blob/1f08d1f3ddc2e7bc7341be4b29bdafaec18de9cc/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java#L70
 

Currently it's 64KB, which is relatively small. Setting it to 1MB or 5MB or so should work around this issue (it should be larger than the maximum row size) - gRPC should support up to 10MB request sizes.

 

Issue Links

• links to: GitHub Pull Request #16838
• links to: GitHub Pull Request #16862