[HDFS-15191] EOF when reading legacy buffer in BlockTokenIdentifier

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.0, 3.2.2
• Component/s: hdfs
• Assignee: Steven Rand
• Reporter: Steven Rand
• Created: 24/Feb/20 20:33
• Updated: 15/Oct/20 08:57
• Resolved: 27/Mar/20 20:02

Description

We have an HDFS client application which recently upgraded from 3.2.0 to 3.2.1. After this upgrade (but not before), we sometimes see these errors when this application is used with clusters still running Hadoop 2.x (more specifically CDH 5.12.1):

WARN  [2020-02-24T00:54:32.856Z] org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing remote block reader. (_sampled: true)
java.io.EOFException:
        at java.io.DataInputStream.readByte(DataInputStream.java:272)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
        at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFieldsLegacy(BlockTokenIdentifier.java:240)
        at org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier.readFields(BlockTokenIdentifier.java:221)
        at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:200)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:530)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:342)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:276)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:245)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:227)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:170)
        at org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:730)
        at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2942)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
        at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:644)
        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:575)
        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:757)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:829)
        at java.io.DataInputStream.read(DataInputStream.java:100)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
        at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765)

We get this warning for all DataNodes with a copy of the block, so the read fails.

I haven't been able to figure out what changed between 3.2.0 and 3.2.1 to cause this, but HDFS-13617 and HDFS-14611 seem related, so tagging vagarychen in case you have any ideas.

Issue Links

• is related to: NIFI-7818 PutHDFS results in EOF due to HDFS-15191