[HIVE-9188] BloomFilter support in ORC

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 1.2.0
• Component/s: File Formats
• Labels: orcfile
• Release Note: Support for Bloom Filters in ORC internal index.
• Assignee: Prasanth Jayachandran
• Reporter: Prasanth Jayachandran
• Created: 22/Dec/14 03:19
• Updated: 17/Jun/15 02:00
• Resolved: 07/Feb/15 09:35

Description

BloomFilters are well known probabilistic data structure for set membership checking. We can use bloom filters in ORC index for better row group pruning. Currently, ORC row group index uses min/max statistics to eliminate row groups (stripes as well) that do not satisfy predicate condition specified in the query. But in some cases, the efficiency of min/max based elimination is not optimal (unsorted columns with wide range of entries). Bloom filters can be an effective and efficient alternative for row group/split elimination for point queries or queries with IN clause.

Issue Links

• is blocked by: HIVE-4639 Add has null flag to ORC internal index
• relates to: HIVE-11033 BloomFilter index is not honored by ORC reader
• relates to: HIVE-9931 Approximate nDV statistics from ORC bloom filter population
• links to: Review Board