Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3791

Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Backport HDFS-173.
      see the comment for more details

      1. HDFS-3791.patch
        19 kB
        Uma Maheswara Rao G
      2. HDFS-3791.patch
        19 kB
        Uma Maheswara Rao G
      3. HDFS-3791.patch
        12 kB
        Suresh Srinivas

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          14d 17h 26m 1 Suresh Srinivas 28/Aug/12 13:00
          Resolved Resolved Closed Closed
          96d 19h 32m 1 Matt Foley 03/Dec/12 07:33
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of 1.1.1.

          Show
          Matt Foley added a comment - Closed upon release of 1.1.1.
          Hide
          Matt Foley added a comment -

          included in branch-1.1

          Show
          Matt Foley added a comment - included in branch-1.1
          Matt Foley made changes -
          Fix Version/s 1.1.1 [ 12321656 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Hide
          Uma Maheswara Rao G added a comment -

          @Ted,
          Yes, I was also in the same lines as Suresh said about that parameter. Not sure about any usecase for getting some advantage by making it configurable.

          Do you have any usecase, where we may get some advatages by tuning that parameter? If yes, feel free to file a JIRA, we can discuss about it there.Thanks a lot, Ted for taking a look on it.

          Thanks,
          Uma

          Show
          Uma Maheswara Rao G added a comment - @Ted, Yes, I was also in the same lines as Suresh said about that parameter. Not sure about any usecase for getting some advantage by making it configurable. Do you have any usecase, where we may get some advatages by tuning that parameter? If yes, feel free to file a JIRA, we can discuss about it there.Thanks a lot, Ted for taking a look on it. Thanks, Uma
          Hide
          Suresh Srinivas added a comment -

          when I added this in trunk, I was not sure if there is a usecase. The whole idea was to give up lock once deleting some number of blocks. So the number currently is arbitrary.

          Show
          Suresh Srinivas added a comment - when I added this in trunk, I was not sure if there is a usecase. The whole idea was to give up lock once deleting some number of blocks. So the number currently is arbitrary.
          Hide
          Ted Yu added a comment -

          Currently small deletion is determined by the constant BLOCK_DELETION_INCREMENT:

          +      deleteNow = collectedBlocks.size() <= BLOCK_DELETION_INCREMENT;
          

          I wonder if there is use case where the increment should be configurable.

          Show
          Ted Yu added a comment - Currently small deletion is determined by the constant BLOCK_DELETION_INCREMENT: + deleteNow = collectedBlocks.size() <= BLOCK_DELETION_INCREMENT; I wonder if there is use case where the increment should be configurable.
          Hide
          Uma Maheswara Rao G added a comment -

          Oh, I have just seen the comments.

          Uma sorry for the delay in reviewing this. +1 for the patch.

          No problem . Thanks a lot, Suresh for the reviews.
          Also thanks for rebasing it. I will to get a patch for HDFS-2815 today in some time.

          Show
          Uma Maheswara Rao G added a comment - Oh, I have just seen the comments. Uma sorry for the delay in reviewing this. +1 for the patch. No problem . Thanks a lot, Suresh for the reviews. Also thanks for rebasing it. I will to get a patch for HDFS-2815 today in some time.
          Suresh Srinivas made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Resolution Fixed [ 1 ]
          Hide
          Suresh Srinivas added a comment -

          I committed the patch. Thank you Uma.

          Show
          Suresh Srinivas added a comment - I committed the patch. Thank you Uma.
          Suresh Srinivas made changes -
          Attachment HDFS-3791.patch [ 12542751 ]
          Hide
          Suresh Srinivas added a comment -

          Rebased the patch on latest branch-1

          Show
          Suresh Srinivas added a comment - Rebased the patch on latest branch-1
          Hide
          Suresh Srinivas added a comment -

          Uma sorry for the delay in reviewing this. +1 for the patch.

          Show
          Suresh Srinivas added a comment - Uma sorry for the delay in reviewing this. +1 for the patch.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Suresh, could you please take a look on this for review?

          Show
          Uma Maheswara Rao G added a comment - Hi Suresh, could you please take a look on this for review?
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks a lot Suresh for the review!
          True, I should have put that inside synchronization block for small deletes. As anyway there should not be harm and going remove that part in side lock with HDFS-2815, I did not take that movement seriously.

          Moved that smaller deletes code inside synchronization. Code wise it should be sync with HDFS-2815 now. as folder structure is diff than trunk it is not possible to apply directly right. SO, I will generate one in HDFS-2815 after you commit this.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Thanks a lot Suresh for the review! True, I should have put that inside synchronization block for small deletes. As anyway there should not be harm and going remove that part in side lock with HDFS-2815 , I did not take that movement seriously. Moved that smaller deletes code inside synchronization. Code wise it should be sync with HDFS-2815 now. as folder structure is diff than trunk it is not possible to apply directly right. SO, I will generate one in HDFS-2815 after you commit this. Thanks Uma
          Uma Maheswara Rao G made changes -
          Attachment HDFS-3791.patch [ 12541231 ]
          Hide
          Suresh Srinivas added a comment -

          Uma, the ported patch looks good. One minor issue - Perform small deletes right away in FSNamesystem.java should move to the synchronized section. Can HDFS-2815 patch be applied on top of this?

          Show
          Suresh Srinivas added a comment - Uma, the ported patch looks good. One minor issue - Perform small deletes right away in FSNamesystem.java should move to the synchronized section. Can HDFS-2815 patch be applied on top of this?
          Uma Maheswara Rao G made changes -
          Link This issue blocks HDFS-2815 [ HDFS-2815 ]
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Suresh, I have just attached a back-port patch here. Could you please take a look.

          Show
          Uma Maheswara Rao G added a comment - Hi Suresh, I have just attached a back-port patch here. Could you please take a look.
          Uma Maheswara Rao G made changes -
          Attachment HDFS-3791.patch [ 12540894 ]
          Uma Maheswara Rao G made changes -
          Summary Backport HDFS-173 Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
          Uma Maheswara Rao G made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-173 [ HDFS-173 ]
          Uma Maheswara Rao G created issue -

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development