Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1143

Implement Background deletion

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 0.22.0
    • None
    • namenode
    • None

    Description

      Right now if you try to delete massive number of files from the namenode it will freeze (sometimes for minutes). Most of the time is spent going through the blocks map and invalidating all the blocks.
      This can probably be improved by having a background GC process. The deletion will basically just remove the inode being deleted and then give the subtree that was just deleted to the background thread running cleanup.
      This way the namenode becomes available for the clients soon after deletion, and all the heavy operations are done in the background.

      Thoughts?

      Attachments

        1. HDFS-1143-v2.txt
          11 kB
          Scott Chen
        2. HDFS-1143.txt
          11 kB
          Scott Chen

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            schen Scott Chen
            dms Dmytro Molkov
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment