[HDFS-14859] Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.3.0, 3.1.4
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Component/s: hdfs
• Labels: block
• Hadoop Flags: Reviewed
• Assignee: Srinivasu Majeti
• Reporter: Srinivasu Majeti
• Due: 24/Sep/19
• Created: 20/Sep/19 06:07
• Updated: 23/Sep/20 19:56
• Resolved: 08/Oct/19 08:51

Description

There have been improvements like HDFS-14171 and HDFS-14632 to the performance issue caused from getNumLiveDataNodes calls per block. The improvement has been only done w.r.t dfs.namenode.safemode.min.datanodes paramter being set to 0 or not.

   private boolean areThresholdsMet() {
     assert namesystem.hasWriteLock();
-    int datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
+    // Calculating the number of live datanodes is time-consuming
+    // in large clusters. Skip it when datanodeThreshold is zero.
+    int datanodeNum = 0;
+    if (datanodeThreshold > 0) {
+      datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
+    }
     synchronized (this) {
       return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
     }

I feel above logic would create similar situation of un-necessary evaluations of getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes paramter is set > 0 even though "blockSafe >= blockThreshold" is false for most of the time in NN startup safe mode. We could do something like below to avoid this

private boolean areThresholdsMet() {
    assert namesystem.hasWriteLock();
    synchronized (this) {
      return blockSafe >= blockThreshold && (datanodeThreshold > 0)?
              blockManager.getDatanodeManager().getNumLiveDataNodes() >= datanodeThreshold : true;
    }
  } 

Issue Links

• relates to: HDFS-15594 Lazy calculate live datanodes in safe mode tip
• supercedes: HDFS-14171 Performance improvement in Tailing EditLog