Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9684

Incorrect disk range computation in ORC because of optional stream kind

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 1.1.0, 1.0.1
    • Fix Version/s: 1.0.0, 1.1.0
    • Component/s: File Formats
    • Labels:
      None

      Description

      HIVE-9593 changed all required fields in ORC protobuf message to optional field. But DiskRange computation and stream creation code assumes existence of stream kind everywhere. This leads to incorrect calculation of diskranges resulting in out of range exceptions. The proper fix is to check if stream kind exists using stream.hasKind() before adding the stream to disk range computation.

      1. HIVE-9684.branch-1.0.patch
        4 kB
        Prasanth Jayachandran
      2. HIVE-9684.branch-1.1.patch
        4 kB
        Prasanth Jayachandran
      3. HIVE-9684.1.patch
        4 kB
        Prasanth Jayachandran

        Activity

        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        Gopal V/Owen O'Malley Can someone review this patch?

        Show
        prasanth_j Prasanth Jayachandran added a comment - Gopal V / Owen O'Malley Can someone review this patch?
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        The issue does not happen in trunk. But the check is required for forward compatibility.

        Show
        prasanth_j Prasanth Jayachandran added a comment - The issue does not happen in trunk. But the check is required for forward compatibility.
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        Attached trunk patch as well.

        Show
        prasanth_j Prasanth Jayachandran added a comment - Attached trunk patch as well.
        Hide
        gopalv Gopal V added a comment -

        LGTM +1.

        This needs the extra condition because unknown enum fields default to the first field (PRESENT).

        Show
        gopalv Gopal V added a comment - LGTM +1. This needs the extra condition because unknown enum fields default to the first field (PRESENT).
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12698799/HIVE-9684.1.patch

        ERROR: -1 due to 2 failed/errored test(s), 7548 tests executed
        Failed tests:

        org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
        org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2797/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2797/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2797/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12698799 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12698799/HIVE-9684.1.patch ERROR: -1 due to 2 failed/errored test(s), 7548 tests executed Failed tests: org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2797/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2797/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2797/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12698799 - PreCommit-HIVE-TRUNK-Build
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        Brock Noland This is a critical issue. Can we have this patch in hive 1.1?

        Show
        prasanth_j Prasanth Jayachandran added a comment - Brock Noland This is a critical issue. Can we have this patch in hive 1.1?
        Hide
        brocknoland Brock Noland added a comment -

        Committed to 1.1.0. I'll let you guys commit to other branches.

        Show
        brocknoland Brock Noland added a comment - Committed to 1.1.0. I'll let you guys commit to other branches.
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        Thanks Brock Noland for committing to 1.1.

        I committed the patch to branch-1.0 and trunk. Thanks Gopal V for the review!

        Show
        prasanth_j Prasanth Jayachandran added a comment - Thanks Brock Noland for committing to 1.1. I committed the patch to branch-1.0 and trunk. Thanks Gopal V for the review!
        Hide
        thejas Thejas M Nair added a comment -

        This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

        Show
        thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          People

          • Assignee:
            prasanth_j Prasanth Jayachandran
            Reporter:
            prasanth_j Prasanth Jayachandran
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development