Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8574

When block count for a volume exceeds dfs.blockreport.split.threshold, block report causes exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 2.7.0
    • None
    • None
    • None

    Description

      This piece of code in org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport()

      // Send one block report per message.
              for (int r = 0; r < reports.length; r++) {
                StorageBlockReport singleReport[] = { reports[r] };
                DatanodeCommand cmd = bpNamenode.blockReport(
                    bpRegistration, bpos.getBlockPoolId(), singleReport,
                    new BlockReportContext(reports.length, r, reportId));
                numReportsSent++;
                numRPCs++;
                if (cmd != null) {
                  cmds.add(cmd);
                }
      

      when a single volume contains many blocks, i.e more than the threshold, it is trying to send the entire blockreport in one RPC, causing exception

      java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
              at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:369)
              at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:347)
              at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:325)
              at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:190)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:473)
      

      Attachments

        Issue Links

          Activity

            People

              ajithshetty Ajith S
              ajithshetty Ajith S
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: