[HUDI-5288] Optimize drop duplicates by avoiding index look up twice

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Fix Version/s: 0.13.0
• Component/s: index
• Labels: pull-request-available
• Story Points: 1
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 28/Nov/22 23:34
• Updated: 21/Dec/22 18:39
• Resolved: 21/Dec/22 18:39

Description

we could potentially optimize DROP_DUPES feature by doing just 1 index look up. 

as of now, we do an explicit index look up and then drop the matched records and proceed on as a regular commit. 

why not proceed on as regular commit and then just after index lookup, we can filter for new inserts and proceed. 

 

Issue Links

• links to: GitHub Pull Request #7303