Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13164

Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      https://github.com/apache/hadoop/blob/27c4e90efce04e1b1302f668b5eb22412e00d033/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1224

      deleteUnnecessaryFakeDirectories is invoked in S3AFileSystem during rename and on outputstream close() to purge any fake directories. Depending on the nesting in the folder structure, it might take a lot longer time as it invokes getFileStatus multiple times. Instead, it should be able to break out of the loop once a non-empty directory is encountered.

        Attachments

        1. HADOOP-13164.branch-2.WIP.patch
          8 kB
          Rajesh Balamohan
        2. HADOOP-13164.branch-2.WIP.002.patch
          12 kB
          Rajesh Balamohan
        3. HADOOP-13164.branch-2-002.patch
          11 kB
          Rajesh Balamohan
        4. HADOOP-13164-branch-2-003.patch
          11 kB
          Rajesh Balamohan
        5. HADOOP-13164-branch-2-004.patch
          11 kB
          Rajesh Balamohan
        6. HADOOP-13164-branch-005.patch
          13 kB
          Steve Loughran

          Issue Links

          causes

          Sub-task - The sub-task of the issue HADOOP-16090 S3A Client to add explicit support for versioned stores

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-13230 S3A to optionally retain directory markers

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14594 ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Assignee:
                rajesh.balamohan Rajesh Balamohan
                Reporter:
                rajesh.balamohan Rajesh Balamohan
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: