Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12740

Reduce and backoff GCS metadata operations when writing to GCS files

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.34.0
    • io-java-gcp
    • None

    Description

      When issuing GCS operations affecting metadata (ie file-level operations not read/write operations), GCS may return errors indicating backoff. See
      https://cloud.google.com/storage/docs/request-rate#ramp-up

      If such errors are encountered, currently the exception is not handled by GcsUtil.java and is propagated, causing retries and backoff of all operations at a higher level. Instead we should backoff and retry only such files that require it.

      Additionally FileBasedSink issues deletes for files that have been renamed. The rename itself should take care of removing the original file and thus we can reduce some metadata operations.

      Attachments

        Issue Links

          Activity

            People

              scwhittle Sam Whittle
              scwhittle Sam Whittle
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 9h 20m
                  9h 20m