Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10941

Improve BlockManager#processMisReplicatesAsync log

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      BlockManager#processMisReplicatesAsync is the daemon thread running inside namenode to handle miserplicated blocks. As shown below, it has a trace log for each of the block in the cluster being processed (10000 blocks per iteration after sleep 10s).

        MisReplicationResult res = processMisReplicatedBlock(block);
        if (LOG.isTraceEnabled()) {
          LOG.trace("block " + block + ": " + res);
        }
      

      However, it is not very useful as dumping every block in the cluster will overwhelm the namenode log without much useful information assuming the majority of the blocks are not over/under replicated. This ticket is opened to improve the log for easy troubleshooting of block replication related issues by:

      1) add debug log for blocks that get under/over replicated result during processMisReplicatedBlock()

      2) or change to trace log for only blocks that get non-OK result during processMisReplicatedBlock()

        Attachments

        1. HDFS-10941.003.patch
          2 kB
          Chen Liang
        2. HDFS-10941.002.patch
          2 kB
          Chen Liang
        3. HDFS-10941.002.patch
          2 kB
          Chen Liang
        4. HDFS-10941.001.patch
          2 kB
          Chen Liang

          Activity

            People

            • Assignee:
              vagarychen Chen Liang
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: