Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-313

Missing checks on SubType count of LIST, MAP and UNION types

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0, 1.1.2, 1.2.3, 1.3.4, 1.4.3
    • 1.5.0
    • C++, Java, Reader
    • None

    Description

      When converting proto::Type to TypeImpl, we need to check that LIST and MAP types have a correct number of subtypes. Otherwise, it will lead to later errors in the c++ reader.

      A file with ill types is attached.

      The java reader also has this problem and throws an IndexOutOfBoundsException immediately:

      Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0
              at java.util.Collections$EmptyList.get(Collections.java:3212)
              at org.apache.orc.OrcProto$Type.getSubtypes(OrcProto.java:12642)
              at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:506)
              at org.apache.orc.OrcUtils.convertTypeFromProtobuf(OrcUtils.java:515)
              at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:386)
              at org.apache.orc.OrcFile.createReader(OrcFile.java:327)
              at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
              at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:300)
              at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
              at org.apache.orc.tools.FileDump.main(FileDump.java:135)
              at org.apache.orc.tools.Driver.main(Driver.java:105)
      

      We should also check the subtype count of UNION type. If its subtype_size == 0, the 'counts' pointer used in UnionColumnReader::skip and UnionColumnReader::next will be null pointer. The attached file (no_subtypes_union.orc) can reproduce this.

      Attachments

        1. no_subtypes_union.orc
          2 kB
          Quanlong Huang
        2. ill_types.orc
          2 kB
          Quanlong Huang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stigahuang Quanlong Huang
            stigahuang Quanlong Huang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment