Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features
  3. HADOOP-15240

ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths failure, FileAlreadyExistsException

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 3.1.0, 3.2.0
    • Fix Version/s: None
    • Component/s: fs/s3, test
    • Labels:
      None

      Description

      Transient ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths failure; FileAlreadyExistsException during the setup phase of the test -writing out the work to
      org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/build/test:fork-0002/job_200707120153_0002/_temporary/1/_temporary/attempt_200707120153_0002_m_000000_0/part-m-00000

      There's something wrong here as that path is a FileOutputCommitter workingDir path, not a partitioned committer.

      Hypotheses: some filesystem caching in the test run is stopping the right committer from being picked up.

      It's possible that this has been happening for a while, but only some cleanup failure from a previous run has caught it.

      Proposed

      • add checks in the AbstractIT test to verify the right committer is picked up in the task context by probing for the working dir: file:// for staging, /__magic/ for magic.
      • and verify that success file is there.

        Attachments

          Activity

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: