Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14171

Performance improvement in Tailing EditLog

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.0.0-alpha1
    • Fix Version/s: 2.10.0, 3.0.4, 3.1.2, 3.3.0, 3.2.1, 2.9.3
    • Component/s: namenode
    • Labels:
      None

      Description

      Stack:

      Thread 456 (Edit log tailer):
      State: RUNNABLE
      Blocked count: 1139
      Waited count: 12
      Stack:
      org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getNumLiveDataNodes(DatanodeManager.java:1259)
      org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode.areThresholdsMet(BlockManagerSafeMode.java:570)
      org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode.checkSafeMode(BlockManagerSafeMode.java:213)
      org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode.adjustBlockTotals(BlockManagerSafeMode.java:265)
      org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.completeBlock(BlockManager.java:1087)
      org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.forceCompleteBlock(BlockManager.java:1118)
      org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.updateBlocks(FSEditLogLoader.java:1126)
      org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:468)
      org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:258)
      org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:161)
      org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:892)
      org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:321)
      org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
      org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$400(EditLogTailer.java:410)
      org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
      org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:414)
      org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
      Thread 455 (pool-16-thread-1):
      
      
      

      code:

      private boolean areThresholdsMet() {
        assert namesystem.hasWriteLock();
        int datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
        synchronized (this) {
          return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
        }
      }
      

      According to the code, each time the method areThresholdsMet() is called, the value of datanodeNum is need to be calculated.  However, in the scenario of datanodeThreshold is equal to 0(0 is the default value of the configuration), This expression datanodeNum >= datanodeThreshold always returns true.

      Calling the method getNumLiveDataNodes() is time consuming at a scale of 10,000 datanode clusters. Therefore, we add the judgment condition, and only when the datanodeThreshold is greater than 0, the datanodeNum is calculated, which improves the perfomance greatly.

      The Call Tree graph is shown in the attached file.

       

        Attachments

        1. HDFS-14171.000.patch
          1 kB
          Kenneth Yang
        2. HDFS-14171_Call-Tree.png
          372 kB
          Kenneth Yang
        3. HDFS-14171.001.patch
          1 kB
          Kenneth Yang
        4. HDFS-14171.002.patch
          1 kB
          Kenneth Yang
        5. HDFS-14171.003.patch
          2 kB
          Kenneth Yang

          Issue Links

            Activity

              People

              • Assignee:
                kennethlnnn Kenneth Yang
                Reporter:
                kennethlnnn Kenneth Yang
              • Votes:
                2 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: