Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Following the S3AFileSystem integration patch in HADOOP-13651, we need to add retry logic.

      In HADOOP-13651, I added TODO comments in most of the places retry loops are needed, including:

      • open(path). If MetadataStore reflects recent create/move of file path, but we fail to read it from S3, retry.
      • delete(path). If deleteObject() on S3 fails, but MetadataStore shows the file exists, retry.
      • rename(src,dest). If source path is not visible in S3 yet, retry.
      • listFiles(). Skip for now. Not currently implemented in S3Guard. I will create a separate JIRA for this as it will likely require interface changes (i.e. prefix or subtree scan).

      We may miss some cases initially and we should do failure injection testing to make sure we're covered. Failure injection tests can be a separate JIRA to make this easier to review.

      We also need basic configuration parameters around retry policy. There should be a way to specify maximum retry duration, as some applications would prefer to receive an error eventually, than waiting indefinitely. We should also be keeping statistics when inconsistency is detected and we enter a retry loop.

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          I've been doing some closure-based wrapping of FS operations in the moved WriteOperationsHelper of HADOOP-13786, in AwsLambda. I think I'd like to put more general AWS retry logic in place here: imagine creating an instance of a hadoop retry policy which could be passed in...the DDB calls would just have a different policy.

          Linking to HADOOP-14012, which proposes TranslateException handling AWS exceptions better. I concur, but it's broader than just DDB, as STS/KMS exceptions get translated badly too ..we're mapping from HTTP status code to File IO errors, when really alternate services may be indicating different issues. There is enough detail inside the exceptions that more specific parsing could handle, something like having a separate translator for AWS exceptions for each specific service, giving them the ability to interpret them themselves.

          Show
          stevel@apache.org Steve Loughran added a comment - I've been doing some closure-based wrapping of FS operations in the moved WriteOperationsHelper of HADOOP-13786 , in AwsLambda . I think I'd like to put more general AWS retry logic in place here: imagine creating an instance of a hadoop retry policy which could be passed in...the DDB calls would just have a different policy. Linking to HADOOP-14012 , which proposes TranslateException handling AWS exceptions better. I concur, but it's broader than just DDB, as STS/KMS exceptions get translated badly too ..we're mapping from HTTP status code to File IO errors, when really alternate services may be indicating different issues. There is enough detail inside the exceptions that more specific parsing could handle, something like having a separate translator for AWS exceptions for each specific service, giving them the ability to interpret them themselves.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          we need to implement retry logic in all AWS calls which bypass the xfer manager, so that transient failures (503/throttle, connection timeout) can get retried. The core code is in the HADOOP-13786 branch; it just needs rollout to the existing methods and policies to deal with s3guard failures: when to fail, when to retry. And, for DDB: when to fall back to the blobstore, which is a different recovery strategy to the rest

          Show
          stevel@apache.org Steve Loughran added a comment - we need to implement retry logic in all AWS calls which bypass the xfer manager, so that transient failures (503/throttle, connection timeout) can get retried. The core code is in the HADOOP-13786 branch; it just needs rollout to the existing methods and policies to deal with s3guard failures: when to fail, when to retry. And, for DDB: when to fall back to the blobstore, which is a different recovery strategy to the rest
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Managed to break tests when working with a bucket whose DDB table was precreated at {{ hadoop s3guard init -write 20 -read 20}} & five parallel test cases

          
          Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 230.809 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
          testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)  Time elapsed: 174.084 sec  <<< ERROR!
          com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: 9NTSEF5S8M3EI7MUN0EV2ERKE3VV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2089)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2065)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeBatchWriteItem(AmazonDynamoDBClient.java:575)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:551)
          	at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
          	at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItemUnprocessed(BatchWriteItemImpl.java:64)
          	at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItemUnprocessed(DynamoDB.java:189)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.processBatchWriteRequest(DynamoDBMetadataStore.java:580)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.prune(DynamoDBMetadataStore.java:761)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Prune.run(S3GuardTool.java:938)
          	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.exec(AbstractS3GuardToolTestBase.java:277)
          	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.exec(AbstractS3GuardToolTestBase.java:255)
          	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommand(AbstractS3GuardToolTestBase.java:194)
          	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommandCLI(AbstractS3GuardToolTestBase.java:206)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Tests run: 62, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 388.583 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
          
          Results :
          
          
          Show
          stevel@apache.org Steve Loughran added a comment - Managed to break tests when working with a bucket whose DDB table was precreated at {{ hadoop s3guard init -write 20 -read 20}} & five parallel test cases Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 230.809 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB) Time elapsed: 174.084 sec <<< ERROR! com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: 9NTSEF5S8M3EI7MUN0EV2ERKE3VV4KQNSO5AEMVJF66Q9ASUAAJG) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2089) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2065) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeBatchWriteItem(AmazonDynamoDBClient.java:575) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:551) at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111) at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItemUnprocessed(BatchWriteItemImpl.java:64) at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItemUnprocessed(DynamoDB.java:189) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.processBatchWriteRequest(DynamoDBMetadataStore.java:580) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.prune(DynamoDBMetadataStore.java:761) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Prune.run(S3GuardTool.java:938) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.exec(AbstractS3GuardToolTestBase.java:277) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.exec(AbstractS3GuardToolTestBase.java:255) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommand(AbstractS3GuardToolTestBase.java:194) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommandCLI(AbstractS3GuardToolTestBase.java:206) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Tests run: 62, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 388.583 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations Results :
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          Note that the DDB throttled EX is 400; you can't use RESTy error code logic here,

          Also, in waitForTableActive() failures are remapped to Illegal Argument Exceptions.

          + that stack trace comes from some code which is meant to be handling throttling. Conclusion: either the logic is wrong or its giving up too early

          Show
          stevel@apache.org Steve Loughran added a comment - - edited Note that the DDB throttled EX is 400; you can't use RESTy error code logic here, Also, in waitForTableActive() failures are remapped to Illegal Argument Exceptions. + that stack trace comes from some code which is meant to be handling throttling. Conclusion: either the logic is wrong or its giving up too early

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              fabbri Aaron Fabbri
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development