[HADOOP-14226] S3Guard: DynamoDBMetadataStore::move() should populate ancestor directories of destination paths

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-13345
• Fix Version/s: HADOOP-13345
• Component/s: fs/s3
• Target Version/s: HADOOP-13345
• Assignee: Mingliang Liu
• Reporter: Mingliang Liu
• Created: 23/Mar/17 23:59
• Updated: 04/Apr/17 18:37
• Resolved: 04/Apr/17 18:37

Description

UPDATE: Instead of changing the test, in this JIRA we make sure for each path to put, DDBMS::move() has records of each directory up to the root. See Aaron Fabbri's comment in this JIRA for more detail.

After running ITestDynamoDBMetadataStoreScale, the test data is not cleaned up. There is a call to clearMetadataStore(ms, count); in the finally clause though. The reason is that, the internally called method DynamoDBMetadataStore::deleteSubtree() is assuming there should be an item for the parent dest path:

parent=/fake-bucket, child=moved-here, is_dir=true

In DynamoDBMetadataStore implementation, we assume that if a path exists, all its ancestors will also exist in the table. We need to pre-create dest path to maintain this invariant so that test data can be cleaned up successfully.

I think there may be other tests with the same problem. Let's identify/address them separately.