Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13204 Über-jira: S3a phase III: scale and tuning
  3. HADOOP-14255

S3A to delete unnecessary fake directory objects in mkdirs()

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha4
    • fs/s3
    • None

    Description

      In S3AFileSystem, as an optimization, we delete unnecessary fake directory objects if that directory contains at least one (nested) file. That is done in closing stream of newly created file. However, if the directory becomes non-empty after we just create an empty subdirectory, we do not delete its fake directory object though that fake directory object becomes "unnecessary".

      So in S3AFileSystem::mkdirs(), we have a pending TODO:

      // TODO: If we have created an empty file at /foo/bar and we then call
      // mkdirs for /foo/bar/baz/roo what happens to the empty file /foo/bar/?
      private boolean innerMkdirs(Path p, FsPermission permission)

      This JIRA is to fix the TODO: provide consistent behavior for a fake directory object between its nested subdirectory and nested file by deleting it.

      See related discussion in HADOOP-14236. Thanks Steve Loughran for discussion.

      Attachments

        1. HADOOP-14255.000.patch
          3 kB
          Mingliang Liu
        2. HADOOP-14255.001.patch
          4 kB
          Mingliang Liu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            liuml07 Mingliang Liu
            liuml07 Mingliang Liu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment