[HUDI-1503] Implement a Hash(Bucket)-based Index

• Type: Wish
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: index, performance
• Epic Link: RFC-29 Bucket Index
• Assignee: Unassigned
• Reporter: Shimin Yang
• Created: 04/Jan/21 08:43
• Updated: 21/Dec/23 15:28

Description

This ticket is to introduce a new hash based index, which can improve the performance of  write operations and speed up the queries at the same time(removing shuffle for Spark/Hive).

The new hash-based index works with a customized hash-based partitioner, which partition records based on the hash value of index keys and a fixed bucket number. So there's no need to visit the existing files to determine which file group each record belongs.

Meanwhile, the file group id, hash mode and bucket num can be used by the query engines to eliminate shuffle introduced by aggregation and join.

We implemented an HoodieIndex based on hive hash function which used on production environment of ByteDance for many very-large volume dataset, and we hope this feature can be contributed to the community soon.