[HDFS-7308] DFSClient write packet size may > 64kB

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.7.0
• Component/s: hdfs-client
• Hadoop Flags: Reviewed
• Assignee: Takuya Fukudome
• Reporter: Tsz-wo Sze
• Created: 29/Oct/14 21:37
• Updated: 24/Apr/15 23:22
• Resolved: 27/Feb/15 15:48

Description

In DFSOutputStream.computePacketChunkSize(..),

  private void computePacketChunkSize(int psize, int csize) {
    final int chunkSize = csize + getChecksumSize();
    chunksPerPacket = Math.max(psize/chunkSize, 1);
    packetSize = chunkSize*chunksPerPacket;
    if (DFSClient.LOG.isDebugEnabled()) {
      ...
    }
  }

We have the following

variables	usual values
psize	dfsClient.getConf().writePacketSize = 64kB
csize	bytesPerChecksum = 512B
getChecksumSize(), i.e. CRC size	32B
chunkSize = csize + getChecksumSize()	544B (not a power of two)
psize/chunkSize	120.47
chunksPerPacket = max(psize/chunkSize, 1)	120
packetSize = chunkSize*chunksPerPacket (not including header)	65280B
PacketHeader.PKT_MAX_HEADER_LEN	33B
actual packet size	65280 + 33 = 65313 < 65536 = 64k

It is fortunate that the usual packet size = 65313 < 64k although the calculation above does not guarantee it always happens (e.g. if PKT_MAX_HEADER_LEN=257, then actual packet size=65537 > 64k.) We should fix the computation in order to guarantee actual packet size < 64k.

Issue Links

• is related to: HDFS-7276 Limit the number of byte arrays used by DFSOutputStream