[BEAM-9822] SpannerIO: Reduce memory usage - especially when streaming

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.20.0, 2.21.0
• Fix Version/s: 2.22.0
• Component/s: io-java-gcp
• Labels: google-cloud-spanner
• Assignee: Niel Markwick
• Reporter: Niel Markwick
• Created: 26/Apr/20 16:26
• Updated: 21/May/20 01:25
• Resolved: 21/May/20 01:25

Description

SpannerIO uses a lot of memory.
In Streaming Dataflow, it uses many times as much (because dataflow creates many worker threads)

Lower the memory use, and change default parameters during streaming to use smaller batches and disable grouping.

Issue Links

• is related to: BEAM-10047 SpannerIO: Combine sorting and batching
• links to: GitHub Pull Request #11529
• links to: GitHub Pull Request #11532
• links to: GitHub Pull Request #11570