[HUDI-992] For hive-style partitioned source data, partition columns synced with Hive will always have String type

• Type: Bug
• Status: In Progress
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: bootstrap, meta-sync
• Story Points: 2
• Epic Link: Improve Bootstrap
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: Udit Mehrotra
• Created: 04/Jun/20 00:54
• Updated: 21/Dec/23 15:33

Description

Currently bootstrap implementation is not able to handle partition columns correctly when the source data has hive-style partitioning, as is also mentioned in https://jira.apache.org/jira/browse/HUDI-915

The schema inferred while performing bootstrap and stored in the commit metadata does not have partition column schema(in case of hive partitioned data). As a result during hive-sync when hudi tries to determine the type of partition column from that schema, it would not find it and assume the default data type string.

Here is where partition column schema is determined for hive-sync:

https://github.com/apache/hudi/blob/master/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveSchemaUtil.java#L417

 

Thus no matter what the data type of partition column is in the source data (atleast what spark infers it as from the path), it will always be synced as string.

 

Issue Links

• is a child of: HUDI-991 Bootstrap Implementation Bugs
• is related to: HUDI-915 Partition Columns missing in files upserted after Metadata Bootstrap