Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15226 Über-JIRA: S3Guard Phase III: Hadoop 3.2 features
  3. HADOOP-15426

Make S3guard client resilient to DDB throttle events and network failures

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.1.0
    • 3.2.0
    • fs/s3
    • None

    Description

      managed to create on a parallel test run

      org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file: com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
      	at 
      
      

      We should be able to handle this. 400 "bad things happened" error though, not the 503 from S3.

      We need a retry handler for DDB throttle operations

      Attachments

        1. HADOOP-15426-001.patch
          42 kB
          Steve Loughran
        2. HADOOP-15426-002.patch
          68 kB
          Steve Loughran
        3. HADOOP-15426-003.patch
          85 kB
          Steve Loughran
        4. HADOOP-15426-004.patch
          84 kB
          Steve Loughran
        5. HADOOP-15426-005.patch
          88 kB
          Steve Loughran
        6. HADOOP-15426-006.patch
          88 kB
          Steve Loughran
        7. HADOOP-15426-007.patch
          88 kB
          Steve Loughran
        8. HADOOP-15426-008.patch
          91 kB
          Steve Loughran
        9. HADOOP-15426-009.patch
          91 kB
          Steve Loughran
        10. HADOOP-15426-010.patch
          92 kB
          Steve Loughran
        11. org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale-output.txt
          37 kB
          Steve Loughran
        12. Screen Shot 2018-07-24 at 15.16.46.png
          113 kB
          Steve Loughran
        13. Screen Shot 2018-07-25 at 16.22.10.png
          47 kB
          Steve Loughran
        14. Screen Shot 2018-07-25 at 16.28.53.png
          24 kB
          Steve Loughran
        15. Screen Shot 2018-07-27 at 14.07.38.png
          94 kB
          Steve Loughran

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stevel@apache.org Steve Loughran
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment