Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3791

Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Backport HDFS-173.
      see the comment for more details

      1. HDFS-3791.patch
        19 kB
        Uma Maheswara Rao G
      2. HDFS-3791.patch
        19 kB
        Uma Maheswara Rao G
      3. HDFS-3791.patch
        12 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Uma Maheswara Rao G added a comment -

          Hi Suresh, I have just attached a back-port patch here. Could you please take a look.

          Show
          Uma Maheswara Rao G added a comment - Hi Suresh, I have just attached a back-port patch here. Could you please take a look.
          Hide
          Suresh Srinivas added a comment -

          Uma, the ported patch looks good. One minor issue - Perform small deletes right away in FSNamesystem.java should move to the synchronized section. Can HDFS-2815 patch be applied on top of this?

          Show
          Suresh Srinivas added a comment - Uma, the ported patch looks good. One minor issue - Perform small deletes right away in FSNamesystem.java should move to the synchronized section. Can HDFS-2815 patch be applied on top of this?
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks a lot Suresh for the review!
          True, I should have put that inside synchronization block for small deletes. As anyway there should not be harm and going remove that part in side lock with HDFS-2815, I did not take that movement seriously.

          Moved that smaller deletes code inside synchronization. Code wise it should be sync with HDFS-2815 now. as folder structure is diff than trunk it is not possible to apply directly right. SO, I will generate one in HDFS-2815 after you commit this.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Thanks a lot Suresh for the review! True, I should have put that inside synchronization block for small deletes. As anyway there should not be harm and going remove that part in side lock with HDFS-2815 , I did not take that movement seriously. Moved that smaller deletes code inside synchronization. Code wise it should be sync with HDFS-2815 now. as folder structure is diff than trunk it is not possible to apply directly right. SO, I will generate one in HDFS-2815 after you commit this. Thanks Uma
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Suresh, could you please take a look on this for review?

          Show
          Uma Maheswara Rao G added a comment - Hi Suresh, could you please take a look on this for review?
          Hide
          Suresh Srinivas added a comment -

          Uma sorry for the delay in reviewing this. +1 for the patch.

          Show
          Suresh Srinivas added a comment - Uma sorry for the delay in reviewing this. +1 for the patch.
          Hide
          Suresh Srinivas added a comment -

          Rebased the patch on latest branch-1

          Show
          Suresh Srinivas added a comment - Rebased the patch on latest branch-1
          Hide
          Suresh Srinivas added a comment -

          I committed the patch. Thank you Uma.

          Show
          Suresh Srinivas added a comment - I committed the patch. Thank you Uma.
          Hide
          Uma Maheswara Rao G added a comment -

          Oh, I have just seen the comments.

          Uma sorry for the delay in reviewing this. +1 for the patch.

          No problem . Thanks a lot, Suresh for the reviews.
          Also thanks for rebasing it. I will to get a patch for HDFS-2815 today in some time.

          Show
          Uma Maheswara Rao G added a comment - Oh, I have just seen the comments. Uma sorry for the delay in reviewing this. +1 for the patch. No problem . Thanks a lot, Suresh for the reviews. Also thanks for rebasing it. I will to get a patch for HDFS-2815 today in some time.
          Hide
          Ted Yu added a comment -

          Currently small deletion is determined by the constant BLOCK_DELETION_INCREMENT:

          +      deleteNow = collectedBlocks.size() <= BLOCK_DELETION_INCREMENT;
          

          I wonder if there is use case where the increment should be configurable.

          Show
          Ted Yu added a comment - Currently small deletion is determined by the constant BLOCK_DELETION_INCREMENT: + deleteNow = collectedBlocks.size() <= BLOCK_DELETION_INCREMENT; I wonder if there is use case where the increment should be configurable.
          Hide
          Suresh Srinivas added a comment -

          when I added this in trunk, I was not sure if there is a usecase. The whole idea was to give up lock once deleting some number of blocks. So the number currently is arbitrary.

          Show
          Suresh Srinivas added a comment - when I added this in trunk, I was not sure if there is a usecase. The whole idea was to give up lock once deleting some number of blocks. So the number currently is arbitrary.
          Hide
          Uma Maheswara Rao G added a comment -

          @Ted,
          Yes, I was also in the same lines as Suresh said about that parameter. Not sure about any usecase for getting some advantage by making it configurable.

          Do you have any usecase, where we may get some advatages by tuning that parameter? If yes, feel free to file a JIRA, we can discuss about it there.Thanks a lot, Ted for taking a look on it.

          Thanks,
          Uma

          Show
          Uma Maheswara Rao G added a comment - @Ted, Yes, I was also in the same lines as Suresh said about that parameter. Not sure about any usecase for getting some advantage by making it configurable. Do you have any usecase, where we may get some advatages by tuning that parameter? If yes, feel free to file a JIRA, we can discuss about it there.Thanks a lot, Ted for taking a look on it. Thanks, Uma
          Hide
          Matt Foley added a comment -

          included in branch-1.1

          Show
          Matt Foley added a comment - included in branch-1.1
          Hide
          Matt Foley added a comment -

          Closed upon release of 1.1.1.

          Show
          Matt Foley added a comment - Closed upon release of 1.1.1.

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development