[HUDI-1573] Spark Sql Writer support Multi preCmp Field

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: spark, spark-sql, writer-core
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: ann
• Created: 02/Feb/21 11:10
• Updated: 07/Sep/22 01:15

Description

HoodieSparkSqlWriter has a param PRECOMBINE_FIELD_OPT_KEY , which specify a nested field.

Now, support getMultiNestedFieldVal process. If PRECOMBINE_FIELD_OPT_KEY has seperator ',' ,  thus multi nested fields to be compared , to solve duplicated input records before tag.

Example : `timestamp`, `gtid` in  binlog source .

Issue Links

• relates to: HUDI-1588 Support multiple ordering fields via payload class config
• relates to: HUDI-1355 Allowing multipleSourceOrdering fields for doing the preCombine on payload
• links to: GitHub Pull Request #2519