[HIVE-13083] Writing HiveDecimal to ORC can wrongly suppress present stream

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0, 0.14.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 2.0.0, 2.1.0
• Fix Version/s: 1.3.0, 2.0.1, 2.1.0
• Target Version/s: 1.3.0, 2.0.1, 2.1.0
• Assignee: Prasanth Jayachandran
• Reporter: Yi Zhang
• Created: 18/Feb/16 06:33
• Updated: 27/Dec/18 23:03
• Resolved: 07/Mar/16 21:09

Description

HIVE-3976 can cause ORC file to be unreadable. The changes introduced in HIVE-3976 for DecimalTreeWriter can create null values after updating the isPresent stream. https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337

As result of the above return statement, isPresent stream state can become wrong. The isPresent stream thinks all values are non-null and hence suppressed. But the data stream will be of 0 length. When reading such files we will get the following exception

Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0
        at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
        at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
        at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
        ... 24 more

Issue Links

• is depended upon by: HIVE-13220 Set isNull[i] = false for non-null decimal values in DecimalColumnVector
• is related to: SPARK-26437 Decimal data becomes bigint to query, unable to query