[HUDI-3782] Update metadata partitions table config when any of them is enabled/disabled

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Labels: pull-request-available
• Story Points: 2
• Epic Link: RFC-27 Multi Modal Indexing
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit
• Created: 02/Apr/22 16:15
• Updated: 12/Apr/22 03:53
• Resolved: 06/Apr/22 09:40

Description

Now that we have table configs for inflight and completed metadata partitions, and given that both reader and writer are going to rely on this config, we need to ensure that this config is updated properly even when any one of the metadata indexes is enabled/disabled.

For instance, let's say user started with metadata enabled and colstats enabed. Colstats partition was fully built and the table config got updated. Now, they disable it and we miss to update table config. So the writers won't update the colstats partition and it would be out of sync. Readers think that the table config has it so it's already in sync but that's incorrect. We need to handle this scenario like how we did for top level metadata enable/disable config in HUDI-3451

Issue Links

• is duplicated by: HUDI-3785 When any specific metadata table partition is disable, ensure to remove the entry from TableConfig
• is related to: HUDI-3451 Add checks for metadata table init to avoid possible out-of-sync
• links to: GitHub Pull Request #5222