Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-794

MergeTask should use COMPRESSRESULT instead of COMPRESSINTERMEDIATE

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0, 0.5.0
    • Fix Version/s: 0.4.0, 0.5.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      HIVE-794. MergeTask should use COMPRESSRESULT. (Saurabh Nanda via zshao)

      Description

      The MergeTask is responsible for merging small output files into bigger files for the final output table.
      The compression settings to be used should be COMPRESSRESULT instead of COMPRESSINTERMEDIATE.

      GenMRFileSink1.java:172:

          FileSinkOperator newOutput = 
            (FileSinkOperator)OperatorFactory.getAndMakeChild(
               new fileSinkDesc(finalName, ts, 
                                parseCtx.getConf().getBoolVar(HiveConf.ConfVars.COMPRESSINTERMEDIATE)),
               fsRS, extract);
      

      Associated mailing list discussion: http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/200908.mbox/%3C794f042d0908122114o7ddb8d18h4f444c1dfa16fa87@mail.gmail.com%3E

        Attachments

        1. HIVE-compression-fix.patch
          0.7 kB
          Saurabh Nanda

          Activity

            People

            • Assignee:
              saurabhnanda Saurabh Nanda
              Reporter:
              zshao Zheng Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: