[HUDI-1468] incremental read support with clustering

• Type: Task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Component/s: incremental-query
• Labels: pull-request-available release-blocker
• Epic Link: Table Services - Clustering
• Assignee: liwei
• Reporter: satish
• Created: 17/Dec/20 20:07
• Updated: 06/Jan/22 23:25
• Resolved: 12/Aug/21 22:42

Description

As part of clustering, metadata such as hoodie_commit_time changes for records that are clustered. This is specific to SparkBulkInsertBasedRunClusteringStrategy implementation. Figure out a way to carry commit_time from original record to support incremental queries.

Also, incremental queries dont work with 'replacecommit' used by clustering HUDI-1264. Change incremental query to work for replacecommits created by Clustering.

Issue Links

• links to: GitHub Pull Request #3211
• links to: GitHub Pull Request #3419