[HUDI-8036] Handle partition schema for custom key gen in SparkHoodieTableFileIndex

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Labels: pull-request-available
• Story Points: 12
• Epic Link: 1.0 Migration
• Assignee: Lokesh Jain
• Reporter: Lokesh Jain
• Created: 31/Jul/24 11:15
• Updated: 2 days ago 19:05
• Resolved: 05/Sep/24 18:43

Description

Currently the partition schema defined for table in SparkHoodieTableFileIndex does not handle the different partition types for the partition columns. These partition types are simple and timestamp for custom based keygen. The Jira aims to handle these partition types and reproduce the issue as mentioned in mentioned in https://github.com/apache/hudi/issues/8343.

Issue Links

• blocks: HUDI-8098 For custom keygen, only require reading of timestamp partition columns
• causes: HUDI-8560 Revert HUDI-8036
• fixes: HUDI-4430 Incorrect type casting while reading HUDI table created with CustomKeyGenerator and unixtimestamp paritioning field
• fixes: HUDI-7902 Partition fields in Table config should store partition field types for custom key generator
• links to: GitHub Pull Request #11710