[HUDI-3844] HoodieIndexer should set existing MDT partitions in props if not already set by user

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Labels: pull-request-available
• Story Points: 2
• Epic Link: Async Metadata Indexing
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit

Reviewers

sivabalan narayanan

• Created: 10/Apr/22 07:15
• Updated: 12/Apr/22 03:00
• Resolved: 12/Apr/22 03:00

Description

Currently, the indexer assumes that only those partitions that are set by user (in props passed to the indexer) are enabled and then goes ahead and deletes the other partitions (except FILES) while fetching metadata writer.

For instance, let's say ingestion writer had metadata enabled (and hence the FILES partition) and also BLOOM_FILTERS index was enabled. Do some commits and it will as usual create files and bloom_filters partition. Now, user wants to create COLUMN_STATS index using the indexer and hence enabled metadata and column_stats index in props passed to the indexer. In this scenario, indexer will presume that only files and column_stats are enabled, while bloom_filters is disabled and the call table.getMetadataWriter() will think bloom_filters needs to be removed, which is wrong. 

Indexer should not presume which indexes (or MDT partitions) are disabled. Instead, it should update its props based on table config. If a partition exists due to regular writers, it should not delete that partition.

Issue Links

• links to: GitHub Pull Request #5293