[HDFS-10941] Improve BlockManager#processMisReplicatesAsync log

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Chen Liang
• Reporter: Xiaoyu Yao
• Created: 30/Sep/16 21:40
• Updated: 22/Nov/16 01:49
• Resolved: 11/Nov/16 22:45

Description

BlockManager#processMisReplicatesAsync is the daemon thread running inside namenode to handle miserplicated blocks. As shown below, it has a trace log for each of the block in the cluster being processed (10000 blocks per iteration after sleep 10s).

  MisReplicationResult res = processMisReplicatedBlock(block);
  if (LOG.isTraceEnabled()) {
    LOG.trace("block " + block + ": " + res);
  }

However, it is not very useful as dumping every block in the cluster will overwhelm the namenode log without much useful information assuming the majority of the blocks are not over/under replicated. This ticket is opened to improve the log for easy troubleshooting of block replication related issues by:

1) add debug log for blocks that get under/over replicated result during processMisReplicatedBlock()

2) or change to trace log for only blocks that get non-OK result during processMisReplicatedBlock()