[BEAM-7549] Provide a better error message for 410 Gone errors from GCS

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: io-java-gcp, io-py-gcp
• Assignee: Unassigned
• Reporter: Chamikara Madhusanka Jayalath
• Created: 13/Jun/19 03:48
• Updated: 04/Jun/22 12:39

Description

For example 410 errors might get raised here.

com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:431)
        at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:289)
        at com.google.cloud.dataflow.sdk.io.FileBasedSink$FileBasedWriter.close(FileBasedSink.java:571)
        at com.google.cloud.dataflow.sdk.io.FileBasedSink$FileBasedWriter.close(FileBasedSink.java:474)
        at com.google.cloud.dataflow.sdk.io.Write$Bound$WriteBundles.finishBundle(Write.java:202)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone

 

Example client side error message that gives a better description to the end-user: "GCS upload of processed data transiently failed and will need to be retried. Some produced data might need to be recomputed"