[HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0-beta2, 1.0.0
• Labels: pull-request-available
• Story Points: 5
• Epic Link: Indexing Functions
• Assignee: Unassigned
• Reporter: Sagar Sumit
• Created: 18/Apr/24 16:26
• Updated: 10/Jun/24 17:56
• Resolved: 15/May/24 13:41

Description

Currently, `HoodieFileIndex` is responsible for partition pruning as well as file skipping. All indexes are being used in lookupCandidateFilesInMetadataTable method through if-else branches. This is not only hard to maintain as we add more indexes, but also induces a static hierarchy. Instead, we need more flexibility so that we can alter logical plan based on availability of indexes. For partition pruning in Spark, we already have HoodiePruneFileSourcePartitions rule but it is injected during the operator optimization batch and it does not modify the result of the LogicalPlan. To be fully extensible, we should be able to rewrite the LogicalPlan. We should be able to inject rules after partition pruning after the operator optimization batch and before any CBO rules that depend on stats. Spark provides injectPreCBORules API to do so, however it is only available in Spark 3.1.0 onwards.

The goal of this ticket is to refactor index hierarchy and create new rules such that Spark version < 3.1.0 still go via the old path, while later versions can modify the plan using an appropriate index and inject as a pre-CBO rule.

Issue Links

• links to: GitHub Pull Request #11074

Sub-Tasks

• Refactor HoodieFileIndex for more flexibility: Closed
• Implement the pre-CBO rules: Closed