[HUDI-1295] Implement: Metadata based bloom index - write path

• Type: Task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.11.0
• Component/s: writer-core
• Labels: pull-request-available
• Story Points: 1
• Epic Link: RFC-27 Multi Modal Indexing
• Assignee: Manoj Govindassamy
• Reporter: Vinoth Chandar

Reviewers

Vinoth Chandar

• Created: 23/Sep/20 05:41
• Updated: 23/Feb/22 18:00
• Resolved: 04/Feb/22 19:38

Description

Idea here to maintain our bloom filters outside of parquet for speedier access from bloom.

 

• Design and impl bloom filter migration to metadata table. 

Design:

schema for the payload: 

key: partitionName_fileName

payload schema:

isDeleted (boolean): true/false

bloom_type: short

ser_bloom: byte[] representing serialized bloom filter. 

 

 

 

Issue Links

• is duplicated by: HUDI-2587 Impl metadata table based bloom index
• links to: GitHub Pull Request #3884
• links to: GitHub Pull Request #3904
• links to: GitHub Pull Request #4352
• links to: GitHub Pull Request #4516
• links to: GitHub Pull Request #4711
• links to: GitHub Pull Request #4740