[HUDI-6188] Unify the logic of intra-partition upsert and cross-partition upsert in flink state index.

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: index
• Labels: pull-request-available
• Story Points: 1
• Assignee: Ying Lin
• Reporter: Ying Lin
• Created: 08/May/23 08:47
• Updated: 25/Jun/23 02:11

Description

Now when partitioning upsert, according to precombine.field parameter, keep the record with the largest value after upserting.

This is widely used to solve the case of out-of-order data, by setting the precombine.field to the event time to keep records with the largest event time.

However, when using the FLINK_STATE index type, if cross-partition occurs, the precombine.field parameter will not fully take effect.

In the case of cross-partitioning, the current logic uses data that arrives later, even if the event time is smaller.

It may be necessary to unify the logic of intra-partition upsert and cross-partition upsert, which is convenient for users to understand and use.

Issue Links

• links to: GitHub Pull Request #8663