[HIVE-794] MergeTask should use COMPRESSRESULT instead of COMPRESSINTERMEDIATE

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0, 0.5.0
• Fix Version/s: 0.4.0, 0.5.0
• Hadoop Flags: Reviewed
• Release Note: HIVE-794. MergeTask should use COMPRESSRESULT. (Saurabh Nanda via zshao)
• Assignee: Saurabh Nanda
• Reporter: Zheng Shao
• Created: 25/Aug/09 08:21
• Updated: 17/Dec/11 00:05
• Resolved: 26/Aug/09 05:59

Description

The MergeTask is responsible for merging small output files into bigger files for the final output table.
The compression settings to be used should be COMPRESSRESULT instead of COMPRESSINTERMEDIATE.

GenMRFileSink1.java:172:

    FileSinkOperator newOutput = 
      (FileSinkOperator)OperatorFactory.getAndMakeChild(
         new fileSinkDesc(finalName, ts, 
                          parseCtx.getConf().getBoolVar(HiveConf.ConfVars.COMPRESSINTERMEDIATE)),
         fsRS, extract);

Associated mailing list discussion: http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/200908.mbox/[e-mail]%3E