Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15191

EOF when reading legacy buffer in BlockTokenIdentifier

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.1
    • Fix Version/s: 3.3.0, 3.2.2
    • Component/s: hdfs
    • Labels:
      None

      Description

      We have an HDFS client application which recently upgraded from 3.2.0 to 3.2.1. After this upgrade (but not before), we sometimes see these errors when this application is used with clusters still running Hadoop 2.x (more specifically CDH 5.12.1):

      WARN  [2020-02-24T00:54:32.856Z] org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing remote block reader. (_sampled: true)
      java.io.EOFException:
              at java.io.DataInputStream.readByte(DataInputStream.java:272)
              at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
              at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
              at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
              at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
              at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
              at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
              at org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
              at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
              at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
              at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
              at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
              at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
              at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
              at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
              at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
              at java.io.DataInputStream.read(DataInputStream.java:100)
              at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
              at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
              at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
              at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
              at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)
      

      We get this warning for all DataNodes with a copy of the block, so the read fails.

      I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging Chen Liang in case you have any ideas.

        Attachments

        1. HDFS-15191-002.patch
          8 kB
          Steven Rand
        2. HDFS-15191-001.patch
          8 kB
          Steven Rand
        3. HDFS-15191.004.patch
          8 kB
          Steven Rand
        4. HDFS-15191.003.patch
          8 kB
          Steven Rand

          Issue Links

            Activity

              People

              • Assignee:
                Steven Rand Steven Rand
                Reporter:
                Steven Rand Steven Rand
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: