Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5445

PacketReceiver populates the packetLen field in PacketHeader incorrectly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 2.2.0
    • Fix Version/s: 2.7.0
    • Component/s: datanode
    • Labels:
    • Environment:

      Ubuntu 12.10, Hadoop 2.1.0-beta

    • Target Version/s:
    • Tags:
      PacketHeader PacketReceiver BlockSender BlockReceiver

      Description

      Summary: PacketReceiver reconstructs PacketHeaders with a packetLen 4 bytes fewer than it should be. It doesn't cause any exceptions because the reconstructed header is never reserialized, and the packetLen field is not used in this part of the code.

      In the BlockSender class, when a Packet is constructed it must be passed the field packetLen, which is defined as the data length, checksum data length, PLUS the length of the packetLen field itself (4 byte integer).

      BlockSender.java
      484:  private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
      485:      boolean transferTo, DataTransferThrottler throttler) throws IOException {
      ...
      491:    int packetLen = dataLen + checksumDataLen + 4;
      ...
      504:    int headerLen = writePacketHeader(pkt, dataLen, packetLen);
      ...
      586:  }
      ...
      792:  private int writePacketHeader(ByteBuffer pkt, int dataLen, int packetLen) {
      793:    pkt.clear();
      794:    // both syncBlock and syncPacket are false
      795:    PacketHeader header = new PacketHeader(packetLen, offset, seqno,
      796:        (dataLen == 0), dataLen, false);
      ...
      802:  }
      

      In the PacketReceiver class, the PacketHeader is reconstructed using the method setFieldsFromData. However, the 4 bytes for the packetLen field length are missing.

      PacketReceiver.java
      112:  private void doRead(ReadableByteChannel ch, InputStream in)
      113:      throws IOException {
      ...
      136:    int payloadLen = curPacketBuf.getInt();
      ...
      144:    int dataPlusChecksumLen = payloadLen - Ints.BYTES;
      ...
      181:    curHeader.setFieldsFromData(dataPlusChecksumLen, headerBuf);
      ...
      192:  }
      

      The solution would be instead to do:

      PacketReceiver.java
      181:    curHeader.setFieldsFromData(payloadLen, headerBuf);
      

      I found this because I was making small modifications to the code that exposed this inconsistency.

        Attachments

        1. HDFS-5445.001.patch
          2 kB
          Kengo Seki

          Activity

            People

            • Assignee:
              JonathanMace Jonathan Mace
              Reporter:
              JonathanMace Jonathan Mace
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified