[HIVE-1133] Refactor InputFormat and OutputFormat for Hive

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: HBase Handler, Serializers/Deserializers
• Assignee: Unassigned
• Reporter: Zheng Shao
• Created: 05/Feb/10 05:38
• Updated: 08/Sep/14 20:31

Description

Currently we ran into several problems of the FileInputFormat/OutputFormat in Hive.

The requirements are:
R1. We want to support HBase: HIVE-806
R2. We want to selectively include files based on file names: HIVE-951
R3. We want to optionally choose to recurse on the directory structure: HIVE-1083
R4. We want to pass the filter condition into the storage (very useful for HBase, and indexed data format)
R5. We want to pass the column selection information into the storage (already done as part of the RCFile, but we can do it better)

We need to structure these requirements and the code structure in a good way to make it extensible.

Issue Links

• blocks: HIVE-806 Hive with HBase as data store to support MapReduce and direct query
• blocks: HIVE-951 Selectively include EXTERNAL TABLE source files via REGEX
• blocks: HIVE-1083 allow sub-directories for an external table/partition
• blocks: HIVE-1222 in metastore, do not store names of inputformat/outputformat/serde for non-native tables
• blocks: HIVE-1226 support filter pushdown against non-native tables
• is related to: HIVE-3660 Improve OutputFormat for Hive
• relates to: HIVE-705 Hive HBase Integration (umbrella)