[BEAM-11006] Allow Failsafe Handling of BigQuery Streaming Writes

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Missing
• Component/s: extensions-java-gcp
• Labels: Clarified Done bigquery google-cloud-bigquery
• Assignee: Unassigned
• Reporter: Dylan Hercher
• Created: 04/Oct/20 23:42
• Updated: 13/Jan/22 21:49
• Resolved: 15/May/21 17:59

Description

To allow handling of a generic failsafe (of any type) would allow a dead letter queue to retain the original source data rather than the cleaned version and could be more easily understood and re-processed.

 

The BigQueryIO.Write currently supports `withFormatFunction` which allows for a serializable function to be applied to each datapoint -> TableRow.  Ideally that same source value could be converted with a separate function:

`withFailsafeFormatFunction` taken (InputT -> TableRow) or possibly (InputT -> OutputT), though the backwards compatibility of OutputT is more difficult.

Issue Links

• links to: GitHub Pull Request #13055