Hive
  1. Hive
  2. HIVE-6002

Create new ORC write version to address the changes to RLEv2

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      HIVE-5994 encodes large negative big integers wrongly. This results in loss of original data that is being written using orc write version 0.12. Bump up the version number to differentiate the bad writes by 0.12 and the good writes by this new version (0.12.1?).

      1. HIVE-6002.2.patch
        2 kB
        Prasanth J
      2. HIVE-6002.1.patch
        0.6 kB
        Prasanth J

        Activity

        Hide
        Owen O'Malley added a comment -

        Rather than introduce a new version, let's add some metadata. add a key name ORC.FIXED.JIRA and make a comma separated list of the fixed jiras. So in this case, HIVE-5994.

        Show
        Owen O'Malley added a comment - Rather than introduce a new version, let's add some metadata. add a key name ORC.FIXED.JIRA and make a comma separated list of the fixed jiras. So in this case, HIVE-5994 .
        Hide
        Prasanth J added a comment -

        These test failures are unrelated and are fixed in trunk now. I ran the same tests in trunk and all tests passed.

        Show
        Prasanth J added a comment - These test failures are unrelated and are fixed in trunk now. I ran the same tests in trunk and all tests passed.
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12619929/HIVE-6002.2.patch

        ERROR: -1 due to 7 failed/errored test(s), 4943 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/964/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/964/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 7 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12619929

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12619929/HIVE-6002.2.patch ERROR: -1 due to 7 failed/errored test(s), 4943 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/964/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/964/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12619929
        Hide
        Prasanth J added a comment -

        Making it as patch available.

        Show
        Prasanth J added a comment - Making it as patch available.
        Hide
        Prasanth J added a comment -

        Gotcha! Fixed that. I did not update ReaderImpl.checkOrcVersion() to reflect the comparison of patch version. If my understanding I correct change in patch versions are compatible with each other as long as major and minor version remains the same (0 and 12 resp. in this case). Please correct me if I am wrong.

        Show
        Prasanth J added a comment - Gotcha! Fixed that. I did not update ReaderImpl.checkOrcVersion() to reflect the comparison of patch version. If my understanding I correct change in patch versions are compatible with each other as long as major and minor version remains the same (0 and 12 resp. in this case). Please correct me if I am wrong.
        Hide
        Owen O'Malley added a comment -

        You can't use [0,121] for the version because it compares them for numeric order. I'd suggest changing the string for [0,12] to "INVALID 0.12" and add a patch level to Version so that you can add [0,12,1].

        Show
        Owen O'Malley added a comment - You can't use [0,121] for the version because it compares them for numeric order. I'd suggest changing the string for [0,12] to " INVALID 0.12" and add a patch level to Version so that you can add [0,12,1] .
        Hide
        Prasanth J added a comment -

        Do we need to discard "0.12" version completely? "0.12" version is not valid anymore. But config option still allows users to specify "0.12" version. In which case, should can we forcefully bump version to "0.12.1"?

        Show
        Prasanth J added a comment - Do we need to discard "0.12" version completely? "0.12" version is not valid anymore. But config option still allows users to specify "0.12" version. In which case, should can we forcefully bump version to "0.12.1"?
        Hide
        Prasanth J added a comment -

        Bumped the ORC write version number to 0.12.1. Owen O'Malley Can you please review this change?

        Show
        Prasanth J added a comment - Bumped the ORC write version number to 0.12.1. Owen O'Malley Can you please review this change?

          People

          • Assignee:
            Prasanth J
            Reporter:
            Prasanth J
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development