[HADOOP-14036] S3Guard: intermittent duplicate item keys failure

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-13345
• Fix Version/s: HADOOP-13345
• Component/s: fs/s3
• Target Version/s: HADOOP-13345
• Hadoop Flags: Reviewed
• Assignee: Mingliang Liu
• Reporter: Aaron Fabbri
• Created: 28/Jan/17 01:07
• Updated: 17/Mar/17 18:03
• Resolved: 17/Mar/17 18:03

Description

I see this occasionally when running integration tests with -Ds3guard -Ddynamo:

testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)  Time elapsed: 2.756 sec  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: move: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
        at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:408)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:869)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:662)
        at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
        at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces

Issue Links

• is depended upon by: HADOOP-13998 Merge initial S3guard release into trunk
• is related to: HADOOP-14170 FileSystemContractBaseTest is not cleaning up test directory clearly