[BEAM-9706] Add ignoreUnknownValues flag to beam.io.WriteToBigQuery

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.36.0
• Component/s: io-py-gcp
• Assignee: Pablo Estrada
• Reporter: Lucas Rolim
• Created: 06/Apr/20 14:50
• Updated: 25/Jan/22 19:24
• Resolved: 25/Jan/22 19:24

Description

This flag is already in the Java SDK and it is critical to deal with ETL that consumes from Pub/Subs populated from plenty of external APIS that I (as data engineer) don't have control.

 

It simply ignores "extra" attributes and doesn't throw an error when inserting on BigQuery.

Java-related issue:
https://issues.apache.org/jira/browse/BEAM-1267

Issue Links

• links to: GitHub Pull Request #16081