[BEAM-3683] Support BigQuery column-based time partitioning

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.4.0
• Component/s: io-java-gcp
• Assignee: Eugene Kirpichov
• Reporter: Eugene Kirpichov
• Created: 10/Feb/18 01:45
• Updated: 03/Sep/21 22:02
• Resolved: 20/Feb/18 21:05

Description

BigQuery now supports tables partitioned by a DATE or TIMESTAMP column. This is very useful for backfilling, because now it doesn't require 1 load job per partition (1 load job for the whole table is fine now), and in case of BigQueryIO.write(), doesn't require using DynamicDestinations - one only needs to specify which field to partition on.

It is specified via TimePartitioning.field: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load (configuration.load.timePartitioning.field).

Seems that the only thing that's needed is to update the BigQuery client - then users can use BigQueryIO.write().withTimePartitioning() in some cases where they previously needed to use write().to(DynamicDestinations).

Plus publicity (e.g. a StackOverflow answer)

CC: reuvenlax chamikara

Issue Links

• links to: GitHub Pull Request #4694