[HDFS-16970] EC: client copy wrong buffer from decode output during pread

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.3.4
• Fix Version/s: 3.2.5
• Component/s: dfsclient, ec, erasure-coding
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: MingHui Luo
• Created: 02/Apr/23 18:27
• Updated: 16/Jan/24 09:02

Description

When dfsStripedInputStream do pread from a striped block group and read internal block timeout, so will read parity block for decode and fill original chunk buffer with decoded data.
Here try to fill original chunk buffer with decoded data, but get wrong data. 
The reason is that
1.original chunk buffer already read some bytes before timeout from blockReader 
2.chunkBytebuffer's slice always fill begin 0 position of decodeByteBuffer 
slice bytebuffer will fill from wrong decodeByteBuffer position, so will get wrong data from pread.

23/03/21 06:31:11 WARN [StripedRead-24] DFSClient: Exception while reading from BP-xxx:blk_-9xxx_xxx of file_xxx from DatanodeInfoWithStorage[10.xxx.xx.xx:50010,DS-xxx,DISK]
java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.xxx.xx.xx:51426 remote=/10.xxx.xx.xx:50010]
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.readChannelFully(PacketReceiver.java:256)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:207)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
    at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.readNextPacket(BlockReaderRemote.java:221)
    at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.read(BlockReaderRemote.java:201)
    at org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:180)
    at org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:172)
    at org.apache.hadoop.hdfs.StripeReader.readToBuffer(StripeReader.java:240)
    at org.apache.hadoop.hdfs.StripeReader.lambda$readCells$0(StripeReader.java:286)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748) 

Issue Links

• links to: GitHub Pull Request #5526