Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
When issuing GCS operations affecting metadata (ie file-level operations not read/write operations), GCS may return errors indicating backoff. See
https://cloud.google.com/storage/docs/request-rate#ramp-up
If such errors are encountered, currently the exception is not handled by GcsUtil.java and is propagated, causing retries and backoff of all operations at a higher level. Instead we should backoff and retry only such files that require it.
Additionally FileBasedSink issues deletes for files that have been renamed. The rename itself should take care of removing the original file and thus we can reduce some metadata operations.
Attachments
Issue Links
- fixes
-
BEAM-12857 Unable to write to GCS due to IndexOutOfBoundsException in FileSystems
- Resolved
- links to