Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-13934

S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to BatchWriteItem limits

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None

      Description

      When using DynamoDBMetadataStore with a insert heavy hive app , it started throwing exceptions in DynamoDBMetadataStore::move. But just with the following exception, it is relatively hard to debug on the real issue in DynamoDB side.

      Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: 1 validation error detected: Value '{ddb-table-name-334=[com.amazonaws.dynamodb.v20120810.WriteRequest@ca1da583, com.amazonaws.dynamodb.v20120810.WriteRequest@ca1fc7cd, com.amazonaws.dynamodb.v20120810.WriteRequest@ca4244e6, com.amazonaws.dynamodb.v20120810.WriteRequest@ca2f58a9, com.amazonaws.dynamodb.v20120810.WriteRequest@ca3525f8,
      ...
      ...
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
              at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
              at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
              at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
              at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
              at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
              at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
              at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
              at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
              at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
              at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:351)
              ... 28 more
      

        Attachments

        1. HADOOP-13934-HADOOP-13345.000.patch
          11 kB
          Mingliang Liu
        2. HADOOP-13934-HADOOP-13345.001.patch
          11 kB
          Mingliang Liu
        3. HADOOP-13934-HADOOP-13345.002.patch
          10 kB
          Mingliang Liu
        4. HADOOP-13934-HADOOP-13345.003.patch
          13 kB
          Mingliang Liu
        5. HADOOP-13934-HADOOP-13345.004.patch
          14 kB
          Mingliang Liu
        6. HADOOP-13934-HADOOP-13345.005.patch
          13 kB
          Mingliang Liu

          Issue Links

            Activity

              People

              • Assignee:
                liuml07 Mingliang Liu
                Reporter:
                rajesh.balamohan Rajesh Balamohan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: