Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1348

Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening. All other requests to NameNode are blocked when this is happening.

      One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

      1. decomissionImp1.patch
        11 kB
        Hairong Kuang
      2. decomissionImp2.patch
        11 kB
        Hairong Kuang
      3. decommission.patch
        10 kB
        Hairong Kuang
      4. decommission1.patch
        10 kB
        Hairong Kuang

        Activity

        Hairong Kuang created issue -
        Hairong Kuang made changes -
        Field Original Value New Value
        Description NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.

        One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.
        NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening. All other requests to NameNode are blocked when this is happening.

        One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.
        Summary DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
        Hairong Kuang made changes -
        Attachment decommission.patch [ 12452862 ]
        Hairong Kuang made changes -
        Attachment decommission1.patch [ 12453640 ]
        Hairong Kuang made changes -
        Attachment decomissionImp1.patch [ 12453809 ]
        Hairong Kuang made changes -
        Attachment decomissionImp2.patch [ 12455885 ]
        Hairong Kuang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Nigel Daley made changes -
        Fix Version/s 0.22.0 [ 12314241 ]
        Allen Wittenauer made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Allen Wittenauer made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Allen Wittenauer made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Allen Wittenauer made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Unresolved [ 9 ]

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development