Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9593

ORC Reader should ignore unknown metadata streams

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0, 0.12.0, 0.13.1, 1.0.0, 1.2.0, 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: File Formats
    • Labels:
      None

      Description

      ORC readers should ignore metadata streams which are non-essential additions to the main data streams.

      This will include additional indices, histograms or anything we add as an optional stream.

      1. HIVE-9593.no-autogen.patch
        1 kB
        Gopal V
      2. hive-9593.patch
        301 kB
        Owen O'Malley

        Activity

        Hide
        owen.omalley Owen O'Malley added a comment -

        This patch changes all of the required fields to be optional. I've gone through the current code to ensure that null pointers from getKind() won't cause NPE.

        Show
        owen.omalley Owen O'Malley added a comment - This patch changes all of the required fields to be optional. I've gone through the current code to ensure that null pointers from getKind() won't cause NPE.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12696902/hive-9593.patch

        ERROR: -1 due to 2 failed/errored test(s), 7478 tests executed
        Failed tests:

        org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
        org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2685/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2685/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2685/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12696902 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12696902/hive-9593.patch ERROR: -1 due to 2 failed/errored test(s), 7478 tests executed Failed tests: org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2685/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2685/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2685/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12696902 - PreCommit-HIVE-TRUNK-Build
        Hide
        gopalv Gopal V added a comment -

        +1 - Confirmed that the change does not affect any encodings on disk.

        During testing, I merged HIVE-9188 + -Pprotobuf to regenerate generated code - that's probably necessary now that JIRA is in.

        /cc: Prasanth Jayachandran

        Show
        gopalv Gopal V added a comment - +1 - Confirmed that the change does not affect any encodings on disk. During testing, I merged HIVE-9188 + -Pprotobuf to regenerate generated code - that's probably necessary now that JIRA is in. /cc: Prasanth Jayachandran
        Hide
        gopalv Gopal V added a comment -

        Rebased to trunk (minus the autogen code) for application to other branches.

        Show
        gopalv Gopal V added a comment - Rebased to trunk (minus the autogen code) for application to other branches.
        Hide
        owen.omalley Owen O'Malley added a comment -

        I committed this. Thanks for the review, Gopal!

        Show
        owen.omalley Owen O'Malley added a comment - I committed this. Thanks for the review, Gopal!
        Hide
        leftylev Lefty Leverenz added a comment -

        Does this need documentation?

        Show
        leftylev Lefty Leverenz added a comment - Does this need documentation?
        Hide
        leftylev Lefty Leverenz added a comment -

        Ping ... any doc needed?

        Show
        leftylev Lefty Leverenz added a comment - Ping ... any doc needed?
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        Lefty Leverenz Not required as this is not user facing.

        Show
        prasanth_j Prasanth Jayachandran added a comment - Lefty Leverenz Not required as this is not user facing.
        Hide
        leftylev Lefty Leverenz added a comment -

        Thanks Prasanth.

        Show
        leftylev Lefty Leverenz added a comment - Thanks Prasanth.

          People

          • Assignee:
            owen.omalley Owen O'Malley
            Reporter:
            gopalv Gopal V
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development