[BEAM-2864] Support backfill deduplication in BigQueryIO.write()

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: io-java-gcp
• Assignee: Unassigned
• Reporter: Eugene Kirpichov
• Created: 07/Sep/17 20:22
• Updated: 03/Jun/22 18:33

Description

See https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/603 motivated by SO question https://stackoverflow.com/questions/46076914/apache-beam-update-bigquery-table-row-with-bigqueryio

Perhaps one way we can do this is make BigQueryIO return a PValue that can be sequenced with other things, and implement a BigQuery.update() transform that executes a single DML statement (or a small collection thereof - since DML in BigQuery is very scarce), and let the user sandwich them together if they would like to.