[HUDI-7902] Partition fields in Table config should store partition field types for custom key generator

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Labels: pull-request-available
• Story Points: 14
• Epic Link: 1.0 Migration
• Assignee: Lokesh Jain
• Reporter: Ethan Guo (this is the old account; please use "yihua")
• Created: 17/Jun/24 23:16
• Updated: 30/Oct/24 00:15
• Resolved: 30/Oct/24 00:15

Description

As of today, writing a partitioned table with CustomKeyGenerator requires write config of the partition field types (they are not stored as a table property).  CustomKeyGenerator requires partition keys to be in the format of field:type (e.g. column1:SIMPLE ). However, only the field names are stored in hoodie.properties file.  We need to store the field types too so that without the write config, the writer configure the correct partition field name and type in Spark datasource and SQL DML writes.

Issue Links

• fixes: HUDI-7280 Add/Drop/Rename table properties hoodie.properties
• is fixed by: HUDI-7996 Store partition type with partition fields in table configs
• is fixed by: HUDI-8036 Handle partition schema for custom key gen in SparkHoodieTableFileIndex
• is fixed by: HUDI-8044 Store timestamp keygen configs for custom key generator
• is fixed by: HUDI-7970 Move methods from HoodieTableConfigUtils to HoodieTableConfig
• links to: GitHub Pull Request #11901