[HADOOP-15420] s3guard ITestS3GuardToolLocal failures in diff tests

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Assignee: Gabor Bota
• Reporter: Aaron Fabbri
• Created: 27/Apr/18 03:17
• Updated: 09/May/18 02:21
• Resolved: 09/May/18 02:01

Description

Noticed this when testing the patch for HADOOP-13756.

 

[ERROR] Failures:

[ERROR]   ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testPruneCommandCLI:221->AbstractS3GuardToolTestBase.testPruneCommand:201->AbstractS3GuardToolTestBase.assertMetastoreListingCount:214->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 Pruned children count [PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/stale; isDirectory=false; length=100; replication=1; blocksize=512; modification_time=1524798258286; access_time=0; owner=hdfs; group=hdfs; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN; isDeleted=false}, PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/fresh; isDirectory=false; length=100; replication=1; blocksize=512; modification_time=1524798262583; access_time=0; owner=hdfs; group=hdfs; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN; isDeleted=false}] expected:<1> but was:<2>

 

Looking through the code, I'm noticing a couple of issues.

 

1. testDiffCommand() is in ITestS3GuardToolLocal, but it should really be running for all MetadataStore implementations.  Seems like it should live in AbstractS3GuardToolTestBase.

2. AbstractS3GuardToolTestBase#createFile() seems wrong. When onMetadataStore is false, it does a ContractTestUtils.touch(file), but the fs is initialized with a MetadataStore present, so seem like the fs will still put the file in the MetadataStore?

There are other tests which explicitly go around the MetadataStore by using fs.setMetadataStore(nullMS), e.g. ITestS3AInconsistency. We should do something similar in AbstractS3GuardToolTestBase#createFile(), minding any issues with parallel test runs.

Issue Links

• relates to: HADOOP-15416 s3guard diff assert failure if source path not found