[HIVE-4816] Followup from HIVE-4551 : Reworking HCat Record Transformation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Sushanth Sowmyan
• Reporter: Sushanth Sowmyan
• Created: 03/Jul/13 23:05
• Updated: 03/Jul/13 23:08

Description

This is a post-bug followup from HIVE-4551, to refactor some HCatalog code to be cleaner/neater. Reposting contents from my previous jira, this jira needs to track the following:

a) HCatRecord and HCatSchema reflect underlying raw data and do no promotions.
b) Introduce a ConversionImpl, which defines various datatype conversion functions, which all default to returning the input, and having a config that allows a user which conversions are implemented.
c) Introduce a PromotedHCatRecord & PromotedHCatSchema that wrap HCatRecord/HCatSchema and use a ConversionImpl.
d) Implement a PigLoaderConversionImpl/PigStorerConversionImpl in hcat-pig-adapter, which implements the following: Short->Int promotion, Short->Int promotion, Boolean->Int promotion
e) Have HCatLoader/HCatStorer use the promoted versions of HCatRecord/HCatSchema which use the PigConversionImpl.
f) Remove the current HCatContext promotion parameters and make them be HCatLoader/HCatStorer parameters.

Issue Links

• relates to: HIVE-4551 HCatLoader smallint/tinyint promotions to Int have issues with ORC integration