[SPARK-12417] Orc bloom filter options are not propagated during file write in spark

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Assignee: Apache Spark
• Reporter: Rajesh Balamohan
• Created: 18/Dec/15 04:43
• Updated: 14/Sep/18 05:51
• Resolved: 10/Sep/18 18:22

Description

ORC bloom filter is supported by the version of hive used in Spark 1.5.2. However, when trying to create orc file with bloom filter option, it does not make use of it.

E.g, following orc output does not create the bloom filter even though the options are specified.

    Map<String, String> orcOption = new HashMap<String, String>();
    orcOption.put("orc.bloom.filter.columns", "*");
    hiveContext.sql("select * from accounts where effective_date='2015-12-30'").write().
        format("orc").options(orcOption).save("/tmp/accounts");

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• is related to: SPARK-25427 Add BloomFilter creation test cases
• relates to: ORC-137 Disable bloomfilter PPD for timestamps for files created before ORC-135
• relates to: ORC-101 Correct the use of the default charset in the bloomfilter
• links to: [Github] Pull Request #10375 (rajeshbalamohan)
• links to: [Github] Pull Request #10842 (rajeshbalamohan)