[HDFS-8734] Erasure Coding: fix one cell need two packets

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-7285
• Hadoop Flags: Reviewed
• Assignee: Walter Su
• Reporter: Walter Su
• Created: 08/Jul/15 10:03
• Updated: 30/Sep/15 19:38
• Resolved: 14/Jul/15 20:12

Description

The default WritePacketSize is 64k
Currently default cellSize is 64k

We hope one cell consumes one packet. In fact it's not.

By default,
chunkSize = 516( 512 data + 4 checksum)
packetSize = 64k
chunksPerPacket = 126 ( See DFSOutputStream#computePacketChunkSize for details)
numBytes of data in one packet = 64512
cellSize = 65536

When first packet is full ( with 64512 data), there are still 65536 - 64512 = 1024 bytes left.

        super.writeChunk(bytes, offset, len, checksum, ckoff, cklen);

        // cell is full and current packet has not been enqueued,
        if (cellFull && currentPacket != null) {
          enqueueCurrentPacketFull();
        }   

When the last 1024 bytes of the cell was written, we meet cellFull and create another packet.