[ORC-313] Missing checks on SubType count of LIST, MAP and UNION types

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.1.2, 1.2.3, 1.3.4, 1.4.3
• Fix Version/s: 1.5.0
• Component/s: C++, Java, Reader
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 03/Mar/18 09:19
• Updated: 14/May/18 22:24
• Resolved: 24/Mar/18 23:18

Description

When converting proto::Type to TypeImpl, we need to check that LIST and MAP types have a correct number of subtypes. Otherwise, it will lead to later errors in the c++ reader.

A file with ill types is attached.

The java reader also has this problem and throws an IndexOutOfBoundsException immediately:

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0
        at java.util.Collections$EmptyList.get(Collections.java:3212)
        at org.apache.orc.OrcProto$Type.getSubtypes(OrcProto.java:12642)
        at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:506)
        at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:515)
        at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:386)
        at org.apache.orc.OrcFile.createReader(OrcFile.java:327)
        at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
        at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:300)
        at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
        at org.apache.orc.tools.FileDump.main(FileDump.java:135)
        at org.apache.orc.tools.Driver.main(Driver.java:105)

We should also check the subtype count of UNION type. If its subtype_size == 0, the 'counts' pointer used in UnionColumnReader::skip and UnionColumnReader::next will be null pointer. The attached file (no_subtypes_union.orc) can reproduce this.

Issue Links

• blocks: ORC-315 More robust on reading random corrupt files
• relates to: ORC-360 Implement error checking on subtype fields in Java
• links to: GitHub Pull Request #225
• links to: GitHub Pull Request #229