Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7424

Retry HTTP 429 errors from GCS w/ exponential backoff when reading data

    XMLWordPrintableJSON

Details

    Description

      This has to be done for both Java and Python SDKs.

      Seems like Java SDK already retries 429 errors w/o backoff (please verify): https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java#L185

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-7667 report GCS throttling time to Dataflow autoscaler

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Triage Needed
          links to

          Web Link GitHub Pull Request #8933

          Web Link GitHub Pull Request #9014

          Activity

            People

              heejong Heejong Lee
              chamikara Chamikara Madhusanka Jayalath
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5h 50m
                  5h 50m