Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15085

Erasure Coding: some ORC data can not be recovery when partial DataNodes are shut down

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: ec
    • Labels:
      None

      Description

      Test environment: hadoop version 3.1.0,  5 datanode

      step to repo:

      1: Set the ec policy RS-3-2-1024k on all of hdfs paths:

      hdfs ec -setPolicy -path / RS-3-2-1024k

      2.Put the small orc file into hdfs on host which is running datanode dn1:

      hdfs dfs -put orcfile /tmp/testec/

      3.Shut down the datanode dn1, and execute the following command to verify the orc data:

      hive --orcfiledump /tmp/testec/orcfile

      4. The error log should be output on the client side:

      Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of length 974814
              at org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:138)
              at org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.<init>(ByteBufferDecodingState.java:48)
              at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
              at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
              at org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:423)
              at org.apache.hadoop.hdfs.PositionStripeReader.decode(PositionStripeReader.java:74)
              at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:382)
              at org.apache.hadoop.hdfs.DFSStripedInputStream.fetchBlockByteRange(DFSStripedInputStream.java:479)
              at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1442)
              at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1400)
              at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
              at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
              at org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:557)
              at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:276)
              at org.apache.orc.impl.RecordReaderImpl.readAllDataStreams(RecordReaderImpl.java:1099)
              at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1055)
              at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1208)
              at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1243)
              at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:273)
              at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:633)
              at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:627)
              at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:309)
              at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
              at org.apache.orc.tools.FileDump.main(FileDump.java:135)
              at org.apache.orc.tools.FileDump.main(FileDump.java:142)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
      
      

        Attachments

        1. orcfile
          2.07 MB
          zhangbutao

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                zhangbutao zhangbutao
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: