[BEAM-13612] When BigqueryIO's STORAGE_API_AT_LEAST_ONCE method is used and the table spec doesn't contain the project id the pipeline fails

• Type: Bug
• Status: Open
• Priority: P2
• Resolution: Unresolved
• Affects Version/s: 2.35.0
• Component/s: io-java-gcp
• Labels: starter
• Assignee: Unassigned
• Reporter: Sergei Lilichenko
• Created: 07/Jan/22 19:35
• Updated: 04/Jun/22 23:24

Description

The validation method correctly uses BigqueryIO's

getTableWithDefaultProject(BigQueryOptions bqOptions), but at the run time the table spec is not checked for absence of the project id, resulting in RuntimeException.

 

 

Stack trace:

Caused by: java.lang.NullPointerException: Required parameter projectId must be specified.
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
    at com.google.api.client.util.Preconditions.checkNotNull(Preconditions.java:138)
    at com.google.api.services.bigquery.Bigquery$Tables$Get.<init>(Bigquery.java:5325)
    at com.google.api.services.bigquery.Bigquery$Tables.get(Bigquery.java:5298)
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.getTable(BigQueryServicesImpl.java:553)
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.getTable(BigQueryServicesImpl.java:542)
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.getTable(BigQueryServicesImpl.java:536)
    at org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinationsTableRow$1.lambda$$0(StorageApiDynamicDestinationsTableRow.java:66)
    at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
    at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
    at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
    at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
    at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
    ... 35 more