[HADOOP-15240] ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths failure, FileAlreadyExistsException

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 3.1.0, 3.2.0
• Component/s: fs/s3, test
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 15/Feb/18 15:48
• Updated: 26/Jul/18 18:30
• Resolved: 26/Jul/18 18:30

Description

Transient ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths failure; FileAlreadyExistsException during the setup phase of the test -writing out the work to
org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/build/test:fork-0002/job_200707120153_0002/_temporary/1/_temporary/attempt_200707120153_0002_m_000000_0/part-m-00000

There's something wrong here as that path is a FileOutputCommitter workingDir path, not a partitioned committer.

Hypotheses: some filesystem caching in the test run is stopping the right committer from being picked up.

It's possible that this has been happening for a while, but only some cleanup failure from a previous run has caught it.

Proposed

• add checks in the AbstractIT test to verify the right committer is picked up in the task context by probing for the working dir: file:// for staging, /__magic/ for magic.
• and verify that success file is there.