[HADOOP-14226] S3Guard: DynamoDBMetadataStore::move() should populate ancestor directories of destination paths

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-13345
• Fix Version/s: HADOOP-13345
• Component/s: fs/s3
• Target Version/s: HADOOP-13345
• Assignee: Mingliang Liu
• Reporter: Mingliang Liu
• Created: 23/Mar/17 23:59
• Updated: 04/Apr/17 18:37
• Resolved: 04/Apr/17 18:37

Description

UPDATE: Instead of changing the test, in this JIRA we make sure for each path to put, DDBMS::move() has records of each directory up to the root. See fabbri's comment in this JIRA for more detail.

After running ITestDynamoDBMetadataStoreScale, the test data is not cleaned up. There is a call to clearMetadataStore(ms, count); in the finally clause though. The reason is that, the internally called method DynamoDBMetadataStore::deleteSubtree() is assuming there should be an item for the parent dest path:

parent=/fake-bucket, child=moved-here, is_dir=true

In DynamoDBMetadataStore implementation, we assume that if a path exists, all its ancestors will also exist in the table. We need to pre-create dest path to maintain this invariant so that test data can be cleaned up successfully.

I think there may be other tests with the same problem. Let's identify/address them separately.