[ORC-536] "expectedEntries should be > 0" error when bloom filters enabled but rowIndexStride set to 0

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.7, 1.6.0
• Component/s: Java
• Assignee: Jason Dere
• Reporter: Jason Dere
• Created: 23/Jul/19 21:48
• Updated: 04/Sep/19 02:00
• Resolved: 02/Aug/19 19:15

Description

java.lang.IllegalArgumentException: expectedEntries should be > 0
    at shadeorcorg.apache.orc.util.BloomFilter.checkArgument(BloomFilter.java:54)
    at shadeorcorg.apache.orc.util.BloomFilter.<init>(BloomFilter.java:63)
    at shadeorcorg.apache.orc.util.BloomFilterUtf8.<init>(BloomFilterUtf8.java:30)
    at shadeorcorg.apache.orc.impl.writer.TreeWriterBase.<init>(TreeWriterBase.java:116)
    at shadeorcorg.apache.orc.impl.writer.StringBaseTreeWriter.<init>(StringBaseTreeWriter.java:64)
    at shadeorcorg.apache.orc.impl.writer.StringTreeWriter.<init>(StringTreeWriter.java:33)
    at shadeorcorg.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:129)
    at shadeorcorg.apache.orc.impl.writer.StructTreeWriter.<init>(StructTreeWriter.java:41)
    at shadeorcorg.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:155)
    at shadeorcorg.apache.orc.impl.writer.StructTreeWriter.<init>(StructTreeWriter.java:41)
    at shadeorcorg.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:155)
    at shadeorcorg.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:180)
    at shadehive.org.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94)
    at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:378)
    at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.initWriter(OrcRecordUpdater.java:603)
    at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.addSimpleEvent(OrcRecordUpdater.java:424)
    at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.addSplitUpdateEvent(OrcRecordUpdater.java:433)
    at shadehive.org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.insert(OrcRecordUpdater.java:485)
    at org.apache.hive.streaming.AbstractRecordWriter.write(AbstractRecordWriter.java:439)
    at org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.write(HiveStreamingConnection.java:794)

Pretty easy to repro with TestWriterImpl:

  public void testNoBFIfNoIndex() throws Exception {
    // overriding the flag should result in a successful write (no exception)
    conf.set(OrcConf.OVERWRITE_OUTPUT_FILE.getAttribute(), "true");
    // Enable bloomfilter, but disable index
    conf.set(OrcConf.ROW_INDEX_STRIDE.getAttribute(), "0");
    conf.set(OrcConf.BLOOM_FILTER_COLUMNS.getAttribute(), "*");
    Writer w = OrcFile.createWriter(testFilePath, OrcFile.writerOptions(conf).setSchema(schema));
    w.close();
  }

cc prasanth_j

Issue Links

• links to: GitHub Pull Request #414