[HUDI-2754] Performance improvement for IncrementalRelation

• Type: Improvement
• Status: Patch Available
• Priority: Critical
• Resolution: Unresolved
• Component/s: incremental-query, performance
• Labels: pull-request-available
• Story Points: 0
• Epic Link: Performance Improvements
• Assignee: Jintao
• Reporter: Jintao

Reviewers

Alexey Kudinkin, Ethan Guo (this is the old account; please use "yihua")

• Created: 12/Nov/21 17:39
• Updated: 21/Dec/23 15:33

Description

When HoodieIncrSource is used to fetch the update from another Hudi table, the IncrementalRelation will be used to read the data. But it has a performance issue because the column pruning and predicate pushdown don't happen. As the result, Hudi reads too much useless data.

By enabling the column pruning and predicate pushdown, the data to read is reduced dramatically.

Issue Links

• links to: GitHub Pull Request #3985