Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6711

ORC maps uses getMapSize() from MapOI which is unreliable

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0, 0.12.0, 0.13.0
    • Fix Version/s: 0.13.0, 0.14.0
    • Component/s: None
    • Labels:

      Description

      HIVE-6707 had issues with map size. getMapSize() of LazyMap and LazyBinaryMap does not deserialize the keys and count the number of unique keys. Since getMapSize() may return non-distinct count of keys, the length of maps stored using ORC's map tree writer will not be in sync with actual map size. As a result of this RLE reader will try to read beyond the disk range expecting more map entries and will throw exception.

      Stack trace will look like:

      Caused by: java.io.EOFException: Read past end of RLE integer from compressed stream Stream for column 2 kind DATA position: 22059699 length: 22059699 range: 0 offset: 22359014 limit: 22359014 range 0 = 0 to 22059699 uncompressed: 53370 to 53370
              at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:54)
              at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:301)
              at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.next(RecordReaderImpl.java:1572)
              at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1330)
              at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$MapTreeReader.next(RecordReaderImpl.java:2041)
              at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1772)
              at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2963)
              at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:121)
      

        Attachments

        1. HIVE-6711.1.patch
          0.9 kB
          Prasanth Jayachandran

          Activity

            People

            • Assignee:
              prasanth_j Prasanth Jayachandran
              Reporter:
              prasanth_j Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: