[BEAM-9834] Spanner IT tests are flaky due to "io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED"

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Cannot Reproduce
• Fix Version/s: Not applicable
• Component/s: io-java-gcp, test-failures
• Labels: flake stale-assigned
• Assignee: Allen Pradeep Xavier
• Reporter: Chamikara Madhusanka Jayalath
• Created: 27/Apr/20 22:08
• Updated: 24/Mar/21 21:15
• Resolved: 15/Jun/20 02:19

Description

For example,

https://builds.apache.org/job/beam_PostCommit_Java/6028/

https://builds.apache.org/job/beam_PostCommit_Java/6025/

https://builds.apache.org/job/beam_PostCommit_Java/6024/

 

https://builds.apache.org/job/beam_PostCommit_Java/6025/testReport/org.apache.beam.sdk.io.gcp.spanner/SpannerReadIT/testQuery/

Error Message

java.util.concurrent.ExecutionException: com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline expired before operation could complete.

Stacktrace

java.util.concurrent.ExecutionException: com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline expired before operation could complete. at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:526) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:507) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)

 

alxavier nielm any idea why calls to Cloud Spanner seems to be timing out recently ?