Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7285 Erasure Coding Support inside HDFS
  3. HDFS-8734

Erasure Coding: fix one cell need two packets

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HDFS-7285
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The default WritePacketSize is 64k
      Currently default cellSize is 64k

      We hope one cell consumes one packet. In fact it's not.

      By default,
      chunkSize = 516( 512 data + 4 checksum)
      packetSize = 64k
      chunksPerPacket = 126 ( See DFSOutputStream#computePacketChunkSize for details)
      numBytes of data in one packet = 64512
      cellSize = 65536

      When first packet is full ( with 64512 data), there are still 65536 - 64512 = 1024 bytes left.

              super.writeChunk(bytes, offset, len, checksum, ckoff, cklen);
      
              // cell is full and current packet has not been enqueued,
              if (cellFull && currentPacket != null) {
                enqueueCurrentPacketFull();
              }   
      

      When the last 1024 bytes of the cell was written, we meet cellFull and create another packet.

        Attachments

        1. HDFS-8734-HDFS-7285.01.patch
          7 kB
          Walter Su
        2. HDFS-8734.01.patch
          0.9 kB
          Walter Su

          Activity

            People

            • Assignee:
              walter.k.su Walter Su
              Reporter:
              walter.k.su Walter Su
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: