Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7980

Incremental BlockReport will dramatically slow down the startup of a namenode

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In the current implementation the datanode will call the reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before calling the bpNamenode.blockReport() method. So in a large(several thousands of datanodes) and busy cluster it will slow down(more than one hour) the startup of namenode.

      List<DatanodeCommand> blockReport() throws IOException {
          // send block report if timer has expired.
          final long startTime = now();
          if (startTime - lastBlockReport <= dnConf.blockReportInterval) {
            return null;
          }
      
          final ArrayList<DatanodeCommand> cmds = new ArrayList<DatanodeCommand>();
      
          // Flush any block information that precedes the block report. Otherwise
          // we have a chance that we will miss the delHint information
          // or we will report an RBW replica after the BlockReport already reports
          // a FINALIZED one.
          reportReceivedDeletedBlocks();
          lastDeletedReport = startTime;
          .........
              // Send the reports to the NN.
          int numReportsSent = 0;
          int numRPCs = 0;
          boolean success = false;
          long brSendStartTime = now();
          try {
            if (totalBlockCount < dnConf.blockReportSplitThreshold) {
              // Below split threshold, send all reports in a single message.
              DatanodeCommand cmd = bpNamenode.blockReport(
                  bpRegistration, bpos.getBlockPoolId(), reports);
      

        Attachments

        1. HDFS-7980-branch-2.6.1.txt
          9 kB
          Vinod Kumar Vavilapalli
        2. HDFS-7980.004.repost.patch
          9 kB
          Walter Su
        3. HDFS-7980.004.patch
          9 kB
          Walter Su
        4. HDFS-7980.003.patch
          8 kB
          Walter Su
        5. HDFS-7980.002.patch
          7 kB
          Walter Su
        6. HDFS-7980.001.patch
          0.9 kB
          Walter Su

          Issue Links

            Activity

              People

              • Assignee:
                walter.k.su Walter Su
                Reporter:
                huizane Hui Zheng
              • Votes:
                0 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: