Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
2.7.0
-
None
-
None
-
None
Description
This piece of code in org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport()
// Send one block report per message. for (int r = 0; r < reports.length; r++) { StorageBlockReport singleReport[] = { reports[r] }; DatanodeCommand cmd = bpNamenode.blockReport( bpRegistration, bpos.getBlockPoolId(), singleReport, new BlockReportContext(reports.length, r, reportId)); numReportsSent++; numRPCs++; if (cmd != null) { cmds.add(cmd); }
when a single volume contains many blocks, i.e more than the threshold, it is trying to send the entire blockreport in one RPC, causing exception
java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit. at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:369) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:347) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:325) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:190) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:473)
Attachments
Issue Links
- is related to
-
HDFS-9011 Support splitting BlockReport of a storage into multiple RPC
- Patch Available