[PARQUET-41] Add bloom filters to parquet statistics

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: format-2.7.0, 1.12.0
• Component/s: parquet-format, parquet-mr
• Labels: filter2 pull-request-available
• Assignee: Junjie Chen
• Reporter: Alex Levenson
• Created: 24/Jul/14 06:28
• Updated: 18/Apr/21 04:11
• Resolved: 26/Feb/20 16:02

Description

For row groups with no dictionary, we could still produce a bloom filter. This could be very useful in filtering entire row groups.
Pull request:
https://github.com/apache/parquet-mr/pull/215

Issue Links

• Blocked: PARQUET-1332 [C++] Add bloom filter utility class
• blocks: HIVE-9260 Implement the bloom filter for the ParquetSerde
• is related to: HIVE-24831 Support writing bloom filters in Parquet
• is related to: SPARK-34562 Leverage parquet bloom filters
• relates to: PARQUET-1979 Optional bloom_filter_offset is filled if no bloom filter is present
• relates to: DRILL-7895 Use Parquet bloom filter
• relates to: PARQUET-1805 Refactor the configuration for bloom filters
• links to: Bloom filter calculations: Filter size by FPP and uniqueness and expected FPP from overloading
• links to: GitHub Pull Request #62
• links to: GitHub Pull Request #99
• links to: GitHub Pull Request #112
• links to: GitHub Pull Request #113
• links to: GitHub Pull Request #425
• links to: GitHub Pull Request #431
• links to: GitHub Pull Request #432
• links to: GitHub Pull Request #484
• links to: GitHub Pull Request #757

Sub-Tasks

• Define the parquet bloom filter statistics in parquet format: Resolved
• [C++] Cross compatibility support with parquet-mr: Open
• [C++] Bloom filter read/write implementation: Open
• [java]Bloom filter read/write implementation: Resolved
• [C++] Integrate Bloom filter into row group filter logic: Open
• [C++] Add bloom filter utility class: Resolved
• Add bloom filter utility class: Resolved
• [C++] replace shared_ptr to unique_ptr in Bloom filter buffer allocation: Open
• [C++] move Bloom filter test binary to parquet-testing repo: Resolved
• [java] Integrate Bloom filter logic: Resolved
• Support nested column Bloom filter: Resolved
• Perform encoding before bloom filters write out: Open
• Store Bloom filters near to footer.: Resolved
• update hash naming of bloom filter: Resolved
• Update parquet thrift to align with spec: Resolved
• Resolve Bloom filter spec concerns: Resolved
• merge bloom filter feature branch to master: Resolved