[HDFS-15240] Erasure Coding: dirty buffer causes reconstruction block error

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.2.2, 3.3.1, 3.4.0
• Component/s: datanode, erasure-coding
• Target Version/s: 3.2.2, 3.3.1, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: HuangTao
• Reporter: HuangTao
• Created: 25/Mar/20 08:19
• Updated: 27/Jan/24 03:27
• Resolved: 08/Dec/20 03:20

Description

1. When read some lzo files we found some blocks were broken.

I read back all internal blocks(b0-b8) of the block group(RS-6-3-1024k) from DN directly, and choose 6(b0-b5) blocks to decode other 3(b6', b7', b8') blocks. And find the longest common sequenece(LCS) between b6'(decoded) and b6(read from DN)(b7'/b7 and b8'/b8).

After selecting 6 blocks of the block group in combinations one time and iterating through all cases, I find one case that the length of LCS is the block length - 64KB, 64KB is just the length of ByteBuffer used by StripedBlockReader. So the corrupt reconstruction block is made by a dirty buffer.

The following log snippet(only show 2 of 28 cases) is my check program output. In my case, I known the 3th block is corrupt, so need other 5 blocks to decode another 3 blocks, then find the 1th block's LCS substring is block length - 64kb.

It means (0,1,2,4,5,6)th blocks were used to reconstruct 3th block, and the dirty buffer was used before read the 1th block.

Must be noted that StripedBlockReader read from the offset 0 of the 1th block after used the dirty buffer.

EDITED for readability.

decode from block[0, 2, 3, 4, 5, 7] to generate block[1', 6', 8']
Check the first 131072 bytes between block[1] and block[1'], the longest common substring length is 4
Check the first 131072 bytes between block[6] and block[6'], the longest common substring length is 4
Check the first 131072 bytes between block[8] and block[8'], the longest common substring length is 4
decode from block[0, 2, 3, 4, 5, 6] to generate block[1', 7', 8']
Check the first 131072 bytes between block[1] and block[1'], the longest common substring length is 65536
CHECK AGAIN: all 27262976 bytes between block[1] and block[1'], the longest common substring length is 27197440  # this one
Check the first 131072 bytes between block[7] and block[7'], the longest common substring length is 4
Check the first 131072 bytes between block[8] and block[8'], the longest common substring length is 4

Now I know the dirty buffer causes reconstruction block error, but how does the dirty buffer come about?

After digging into the code and DN log, I found this following DN log is the root reason.

[INFO] [stripedRead-1017] : Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=/xxxxxxxx:52586 remote=/xxxxxxxx:50010]. 180000 millis timeout left.
[WARN] [StripedBlockReconstruction-199] : Failed to reconstruct striped block: BP-714356632-xxxxxxxx-1519726836856:blk_-YYYYYYYYYYYYYY_3472979393
java.lang.NullPointerException
    at org.apache.hadoop.hdfs.util.StripedBlockUtil.getNextCompletedStripedRead(StripedBlockUtil.java:314)
    at org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.doReadMinimumSources(StripedReader.java:308)
    at org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.readMinimumSources(StripedReader.java:269)
    at org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.reconstruct(StripedBlockReconstructor.java:94)
    at org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:60)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834) 

Reading from DN may timeout(hold by a future(F)) and output the INFO log, but the futures that contains the future(F)  is cleared, 

return new StripingChunkReadResult(futures.remove(future),
    StripingChunkReadResult.CANCELLED); 

futures.remove(future) cause NPE. So the EC reconstruction is failed. In the finally phase, the code snippet in getStripedReader().close() 

reconstructor.freeBuffer(reader.getReadBuffer());
reader.freeReadBuffer();
reader.closeBlockReader(); 

free buffer firstly, but the StripedBlockReader still holds the buffer and write it, that pollute the buffer of BufferPool.