[HCATALOG-237] Switch from using StorageDrivers to SerDes to do data (de)serialization

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4
• Fix Version/s: 0.4
• Assignee: Unassigned
• Reporter: Alan Gates
• Created: 27/Jan/12 21:08
• Updated: 17/May/12 01:20
• Resolved: 19/Mar/12 15:44

Description

HCatalog started by creating its own classes, InputStorageDriver and OutputStorageDriver, to do data conversion between the storage layer Input/OutputFormats and the HCatInput/OutputFormats. These provide very similar functionality to Hive's SerDe class, though with a much simpler interface.

This usage of separate classes has led to a number of issues for HCatalog. One, it cannot make use of existing Hive SerDes. Two, it has led to a need to make HCat specific extensions of Hive interfaces (such as the StorageHandler) to provide the StorageDescriptors. Three, it means that users who already have Hive installed cannot use HCatalog without first updating every partition in their metastore with storage driver information.

I propose we switch to using SerDes for this. To address the issue of the more complicated SerDe interface we can provide adaptor classes that make writing new SerDes easy in simple cases.

Issue Links

• breaks: HCATALOG-277 JSON reads and writes fail
• breaks: HCATALOG-276 After merging in HCATALOG-237 related changes Pig scripts with more than one store fail
• incorporates: HIVE-2768 Add a getAuthorizationProvider to HiveStorageHandler

Sub-Tasks

• LazyHCatTuple introduction to prevent paying full cost of deserialization of LazyHCatRecord: Open
• only serialize OutputJobInfo into tableDesc.getJobProperties() when calling configureOutputJobProperties(): Reopened
• move setInputPath to FosterStorageHandler.configureInputProperties(): Open