[HUDI-1611] Allow directories to be filtered during the bootstrap of the metadata table

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Resolved
• Fix Version/s: 0.8.0
• Labels: pull-request-available
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Prashant Wason
• Reporter: Prashant Wason
• Created: 10/Feb/21 23:02
• Updated: 04/Jan/22 00:09
• Resolved: 23/Mar/21 14:21

Description

During the bootstrap of the Metadata Table, all the directories which contain the partition metadata directory are assumed to be partitions and are added to the metadata table.

In our HDFS clusters, we have directories like .backup, .temp which are used by various teams for non-hoodie purposes (e.g. .backup may be keeping a snapshot of the dataset). During bootstrap, Metadata Table ends up containing all those paths also as partitions.

In this patch, I would like to introduce a configuration for HoodieMetadataConfig to filter out some directories based on a regular expression string. 

Issue Links

• links to: GitHub Pull Request #2565