Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: File Formats
    • Labels:
      None

      Description

      It would be good to add a bzip compressor for ORC. Bzip does very well for long term/cold storage.

        Activity

        Hide
        Zhichun Wu added a comment -

        I would like to test orc + bzip2 for cold data compression and have a quick imp with BZip2Compressor in apache common compress based on SnappyCodec.java . Right now I just implement the CompressionCodec interface only.

        According to my test, orc + bzip2 doesn't have significant improvement on data compression compared to orc + zlib and it takes more time to complete the compression. Please correct me if I'm wrong.

        Show
        Zhichun Wu added a comment - I would like to test orc + bzip2 for cold data compression and have a quick imp with BZip2Compressor in apache common compress based on SnappyCodec.java . Right now I just implement the CompressionCodec interface only. According to my test, orc + bzip2 doesn't have significant improvement on data compression compared to orc + zlib and it takes more time to complete the compression. Please correct me if I'm wrong.
        Hide
        Gopal V added a comment -

        Try increasing orc.compress.size to the bzip2's 900kb block size.

        Show
        Gopal V added a comment - Try increasing orc.compress.size to the bzip2's 900kb block size.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12638522/HIVE-5067.patch

        ERROR: -1 due to 1 failed/errored test(s), 5548 tests executed
        Failed tests:

        org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testExecuteStatementAsync
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2122/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2122/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12638522

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12638522/HIVE-5067.patch ERROR: -1 due to 1 failed/errored test(s), 5548 tests executed Failed tests: org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testExecuteStatementAsync Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2122/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2122/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12638522

          People

          • Assignee:
            Unassigned
            Reporter:
            Owen O'Malley
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development