Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13164

Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      https://github.com/apache/hadoop/blob/27c4e90efce04e1b1302f668b5eb22412e00d033/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1224

      deleteUnnecessaryFakeDirectories is invoked in S3AFileSystem during rename and on outputstream close() to purge any fake directories. Depending on the nesting in the folder structure, it might take a lot longer time as it invokes getFileStatus multiple times. Instead, it should be able to break out of the loop once a non-empty directory is encountered.

        Attachments

        1. HADOOP-13164-branch-005.patch
          13 kB
          Steve Loughran
        2. HADOOP-13164-branch-2-004.patch
          11 kB
          Rajesh Balamohan
        3. HADOOP-13164-branch-2-003.patch
          11 kB
          Rajesh Balamohan
        4. HADOOP-13164.branch-2-002.patch
          11 kB
          Rajesh Balamohan
        5. HADOOP-13164.branch-2.WIP.002.patch
          12 kB
          Rajesh Balamohan
        6. HADOOP-13164.branch-2.WIP.patch
          8 kB
          Rajesh Balamohan

        Issue Links

        causes

        Sub-task - The sub-task of the issue HADOOP-16090 S3A Client to add explicit support for versioned stores

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        is depended upon by

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance

        • Major - Major loss of function.
        • Resolved
        is related to

        Sub-task - The sub-task of the issue HADOOP-13230 S3A to optionally retain directory markers

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-14594 ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions

        • Major - Major loss of function.
        • Resolved

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              rajesh.balamohan Rajesh Balamohan
              Reporter:
              rajesh.balamohan Rajesh Balamohan

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment