[HADOOP-16207] Improved S3A MR tests

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: fs/s3, test
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 26/Mar/19 13:07
• Updated: 04/Oct/19 13:27
• Resolved: 04/Oct/19 13:17

Description

Reported failure of ITestDirectoryCommitMRJob in validation runs of HADOOP-16186; assertIsDirectory with s3guard enabled and a parallel test run: Path "is recorded as deleted by S3Guard"

    waitForConsistency();
    assertIsDirectory(outputPath) /* here */

The file is there but there's a tombstone. Possibilities

• some race condition with another test
• tombstones aren't timing out
• committers aren't creating that base dir in a way which cleans up S3Guard's tombstones.

Remember: we do have to delete that dest dir before the committer runs unless overwrite==true, so at the start of the run there will be a tombstone. It should be overwritten by a success.

Issue Links

• blocks: HADOOP-14936 S3Guard: remove "experimental" from documentation
• depends upon: MAPREDUCE-6521 MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
• is blocked by: HADOOP-16184 S3Guard: Handle OOB deletions and creation of a file which has a tombstone marker
• is duplicated by: HADOOP-16422 AbstractITCommitMRJob.testMRJob test failures
• is related to: HADOOP-16415 Speed up S3A test runs
• is related to: HADOOP-16186 S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren
• links to: GitHub Pull Request #1115