[HDFS-15085] Erasure Coding: some ORC data can not be recovery when partial DataNodes are shut down

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.1.0
• Component/s: ec
• Assignee: Unassigned
• Reporter: Butao Zhang
• Created: 30/Dec/19 08:12
• Updated: 30/Dec/19 10:04
• Resolved: 30/Dec/19 10:04

Description

Test environment: hadoop version 3.1.0,  5 datanode

step to repo:

1: Set the ec policy RS-3-2-1024k on all of hdfs paths:

hdfs ec -setPolicy -path / RS-3-2-1024k

2.Put the small orc file into hdfs on host which is running datanode dn1:

hdfs dfs -put orcfile /tmp/testec/

3.Shut down the datanode dn1, and execute the following command to verify the orc data:

hive --orcfiledump /tmp/testec/orcfile

4. The error log should be output on the client side:

Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of length 974814
        at org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:138)
        at org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.<init>(ByteBufferDecodingState.java:48)
        at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
        at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
        at org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:423)
        at org.apache.hadoop.hdfs.PositionStripeReader.decode(PositionStripeReader.java:74)
        at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:382)
        at org.apache.hadoop.hdfs.DFSStripedInputStream.fetchBlockByteRange(DFSStripedInputStream.java:479)
        at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1442)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1400)
        at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
        at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
        at org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:557)
        at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:276)
        at org.apache.orc.impl.RecordReaderImpl.readAllDataStreams(RecordReaderImpl.java:1099)
        at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1055)
        at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1208)
        at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1243)
        at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:273)
        at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:633)
        at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:627)
        at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:309)
        at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
        at org.apache.orc.tools.FileDump.main(FileDump.java:135)
        at org.apache.orc.tools.FileDump.main(FileDump.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:222)

Issue Links

• duplicates: HDFS-14373 EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe
• relates to: HDFS-14373 EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe