[HIVE-1950] Block merge for RCFile

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Hadoop Flags: Reviewed
• Assignee: He Yongqiang
• Reporter: He Yongqiang
• Created: 03/Feb/11 18:39
• Updated: 07/Jul/15 07:38
• Resolved: 23/Feb/11 23:59

Description

In our env, there are a lot of small files inside one partition/table. In order to reduce the namenode load, we have one dedicated housekeeping job running to merge these file. Right now the merge is an 'insert overwrite' in hive, and requires decompress the data and compress it. This jira is to add a command in Hive to do the merge without decompress and recompress the data.

Something like "alter table tbl_name [partition ()] concatenate". In this jira the new command will only support RCFile, since there need some new APIs to the fileformat.

Issue Links

• is related to: HIVE-1343 add an interface in RCFile to support concatenation of two files without (de)compression
• is related to: HIVE-2035 Use block-level merge for RCFile if merging intermediate results are needed
• relates to: HIVE-7509 Fast stripe level merging for ORC