[HIVE-7509] Fast stripe level merging for ORC

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.0
• Labels: orcfile

Release Note

Added hive.merge.orcfile.stripe.level configuration parameter here https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveConfigurationProperties

Added concatenate syntax to https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionConcatenate

• Assignee: Prasanth Jayachandran Assign to me
• Reporter: Prasanth Jayachandran
• Created: 25/Jul/14 02:30
• Updated: 15/Mar/16 03:33
• Resolved: 31/Jul/14 03:54

Description

Similar to HIVE-1950, add support for fast stripe level merging of ORC files through CONCATENATE command and conditional merge task. This fast merging is ideal for merging many small ORC files to a larger file without decompressing and decoding the data of small orc files.

Issue Links

• is related to: HIVE-7704 Create tez task for fast file merging
• is related to: HIVE-1343 add an interface in RCFile to support concatenation of two files without (de)compression
• is related to: HIVE-1950 Block merge for RCFile
• is related to: HIVE-2035 Use block-level merge for RCFile if merging intermediate results are needed
• relates to: HIVE-13285 Orc concatenation may drop old files from moving to final path
• links to: Review Board

Sub-Tasks

• Precondition checks should not fail the merge task in case of automatic trigger: Closed