[BEAM-9745] [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Later
• Fix Version/s: Not applicable
• Component/s: io-java-gcp, java-fn-execution, sdk-java-harness, test-failures
• Labels: Done currently-failing
• Assignee: Kenneth Knowles
• Reporter: Daniel Oliveira
• Created: 13/Apr/20 19:11
• Updated: 06/Oct/20 21:56
• Resolved: 29/Jun/20 19:30

Description

Use this form to file an issue for test failure:

• Jenkins Job
• Gradle Build Scan

Initial investigation:

The bug appears to be popping up on BigQuery tests mostly, but also a BigTable and a Datastore test.

Here's an example stacktrace of the two errors, showing only the error messages themselves. Source: https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe

java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error received from SDK harness for instruction -191: java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With Execution Info
...
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for instruction -191: java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With Execution Info
...
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3



java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error received from SDK harness for instruction -206: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for instruction -206: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
...

Update: Looks like this has been failing as far back as Apr 4 after a long period where the test was consistently timing out since Mar 31. So it's hard to narrow down what commit may have caused this. Plus, the test was failing due to a completely different BigQuery failure before anyway, so it seems like this test will need to be completely fixed from scratch, instead of tracking down a specific breaking change.

---

After you've filled out the above details, please assign the issue to an individual. Assignee should treat test failures as high-priority, helping to fix the issue or find a more appropriate owner. See Apache Beam Post-Commit Policies.

Issue Links

• is related to: BEAM-9390 [PostCommit_Java_PortabilityApi] [BigQuery related ITs] UnsupportedOperationException: BigQuery source must be split before being read
• is related to: BEAM-9868 All tests in beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow started to fail since 04/17/2020