[BEAM-14035] Convert BigQuery SchemaIO to SchemaTransform

• Type: Sub-task
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-java-core
• Assignee: Damon Douglas
• Reporter: Lara Schmidt
• Created: 02/Mar/22 21:50
• Updated: 17/May/22 13:58
• Resolved: 16/May/22 18:07

Description

The output of this task is to refactor org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider [1] to implement SchemaTransform [2] and SchemaTransformProvider interfaces [3].

Please see https://issues.apache.org/jira/browse/BEAM-14168 for more information on additional work remaining after this task is complete.

As a result of this task there will be:
1. one class deletion and four classes created for the PR reflecting whether we are reading from or writing to BigQuery, via BigQueryIO.Read and BigQueryIO.Write, respectively.

• delete: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaIOProvider.java
• create: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaTransformReadConfiguration.java
• create: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaTransformReadProvider.java
• create: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaTransformWriteConfiguration.java
• create: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaTransformWriteProvider.java

2. The testing strategy will leverage the use of the org.apache.beam.sdk.transforms.display.DisplayData class instead of using the org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices.
We will test whether the input of BigQuerySchemaTransformReadConfiguration yields a BigQueryIO.Read with correct query, tableSpec, useLegacySQL (defaulted to false), and queryLocation values.
We will test whether the input of BigQuerySchemaTransformWriteConfiguration yields a BigQueryIO.Write with correct tableReference, createDisposition, and writeDisposition values.

References:

1 - sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaIOProvider.java
2 - SchemaTransform
3 - SchemaTransformProvider

Issue Links

• is a child of: BEAM-14021 Convert usage of SchemaIO to SchemaTransform
• links to: GitHub Pull Request #17135
• links to: GitHub Pull Request #17181
• links to: GitHub Pull Request #17607
• links to: GitHub Pull Request #17692