Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-536

"expectedEntries should be > 0" error when bloom filters enabled but rowIndexStride set to 0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.7, 1.6.0
    • Component/s: Java
    • Labels:
      None

      Description

      java.lang.IllegalArgumentException: expectedEntries should be > 0
          at shadeorcorg.apache.orc.util.BloomFilter.checkArgument(BloomFilter.java:54)
          at shadeorcorg.apache.orc.util.BloomFilter.<init>(BloomFilter.java:63)
          at shadeorcorg.apache.orc.util.BloomFilterUtf8.<init>(BloomFilterUtf8.java:30)
          at shadeorcorg.apache.orc.impl.writer.TreeWriterBase.<init>(TreeWriterBase.java:116)
          at shadeorcorg.apache.orc.impl.writer.StringBaseTreeWriter.<init>(StringBaseTreeWriter.java:64)
          at shadeorcorg.apache.orc.impl.writer.StringTreeWriter.<init>(StringTreeWriter.java:33)
          at shadeorcorg.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:129)
          at shadeorcorg.apache.orc.impl.writer.StructTreeWriter.<init>(StructTreeWriter.java:41)
          at shadeorcorg.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:155)
          at shadeorcorg.apache.orc.impl.writer.StructTreeWriter.<init>(StructTreeWriter.java:41)
          at shadeorcorg.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:155)
          at shadeorcorg.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:180)
          at shadehive.org.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94)
          at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:378)
          at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.initWriter(OrcRecordUpdater.java:603)
          at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.addSimpleEvent(OrcRecordUpdater.java:424)
          at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.addSplitUpdateEvent(OrcRecordUpdater.java:433)
          at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.insert(OrcRecordUpdater.java:485)
          at org.apache.hive.streaming.AbstractRecordWriter.write(AbstractRecordWriter.java:439)
          at org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.write(HiveStreamingConnection.java:794)
      

      Pretty easy to repro with TestWriterImpl:

        public void testNoBFIfNoIndex() throws Exception {
          // overriding the flag should result in a successful write (no exception)
          conf.set(OrcConf.OVERWRITE_OUTPUT_FILE.getAttribute(), "true");
          // Enable bloomfilter, but disable index
          conf.set(OrcConf.ROW_INDEX_STRIDE.getAttribute(), "0");
          conf.set(OrcConf.BLOOM_FILTER_COLUMNS.getAttribute(), "*");
          Writer w = OrcFile.createWriter(testFilePath, OrcFile.writerOptions(conf).setSchema(schema));
          w.close();
        }
      

      cc Prasanth Jayachandran

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jdere Jason Dere
                Reporter:
                jdere Jason Dere
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m