[BEAM-13268] Reduce latency by parallelizing BQ inserts when flushing due to row limit

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.35.0
• Component/s: io-java-gcp
• Assignee: Sam Whittle
• Reporter: Sam Whittle
• Created: 17/Nov/21 11:18
• Updated: 17/Nov/21 19:54
• Resolved: 17/Nov/21 19:54

Description

InsertBatchedElements consumes the output of GroupIntoBatches which flushes after default 500 elements to respect the max items per streaming insert request to BQ.

However InsertBatchedElements flushes rows synchrounously meaning that latencies of writes accumulates. It could instead initiate writes in ProcessElement and block on write completion in FinishBundle. There could be some limited configurable parallelism if desired to limit memory usage.

Issue Links

• links to: GitHub Pull Request #16003