Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12363

Incorrect results with orc ppd across ORC versions

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.3.0, 2.0.0
    • 1.3.0, 2.0.0
    • None
    • None
    • Hide
      "HIVE-12363: Incorrect results with orc ppd across ORC versions (Gopal V, reviewed by Prasanth Jayachandran)"
      Show
      " HIVE-12363 : Incorrect results with orc ppd across ORC versions (Gopal V, reviewed by Prasanth Jayachandran)"

    Description

      Run vector_decimal_cast.q on tez cli driver.

      The issue is related to the ORC Timestamp column stats, which does not exist in all ORC files.

      When the timestamp column is missing stats, default to YES_NO_NULL instead of assuming the column is all nulls.

      Attachments

        1. HIVE-12363.branch-1.patch
          1 kB
          Gopal Vijayaraghavan
        2. HIVE-12363.1.patch
          1 kB
          Gopal Vijayaraghavan

        Issue Links

          Activity

            mvn test -Dtest=TestMiniTezCliDriver -Dtest.output.overwrite=true -Dqfile=vector_decimal_cast.q -Dhive.optimize.index.filter=true
            

            and you get this in diffs:

            Begin query: vector_decimal_cast.q
            Running: diff -a /Users/ashutosh/workspace/eclipse-workspace/hive/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/vector_decimal_cast.q.out /Users/ashutosh/workspace/eclipse-workspace/hive/itests/qtest/../../ql/src/test/results/clientpositive/tez/vector_decimal_cast.q.out
            15d14
            <           filterExpr: (((cdouble is not null and cint is not null) and cboolean1 is not null) and ctimestamp1 is not null) (type: boolean)
            32a32,41
            > -13326.0	528534767	true	1969-12-31 15:59:46.674	-13326.0000000000	528534767.00000000000000	1.00	-13
            > -15813.0	528534767	true	1969-12-31 15:59:55.787	-15813.0000000000	528534767.00000000000000	1.00	-4
            > -9566.0	528534767	true	1969-12-31 15:59:44.187	-9566.0000000000	528534767.00000000000000	1.00	-16
            > 15007.0	528534767	true	1969-12-31 15:59:50.434	15007.0000000000	528534767.00000000000000	1.00	-10
            > 7021.0	528534767	true	1969-12-31 16:00:15.007	7021.0000000000	528534767.00000000000000	1.00	15
            > 4963.0	528534767	true	1969-12-31 16:00:07.021	4963.0000000000	528534767.00000000000000	1.00	7
            > -7824.0	528534767	true	1969-12-31 16:00:04.963	-7824.0000000000	528534767.00000000000000	1.00	5
            > -15431.0	528534767	true	1969-12-31 15:59:52.176	-15431.0000000000	528534767.00000000000000	1.00	-8
            > -15549.0	528534767	true	1969-12-31 15:59:44.569	-15549.0000000000	528534767.00000000000000	1.00	-15
            > 5780.0	528534767	true	1969-12-31 15:59:44.451	5780.0000000000	528534767.00000000000000	1.00	-16
            
            ashutoshc Ashutosh Chauhan added a comment - mvn test -Dtest=TestMiniTezCliDriver -Dtest.output.overwrite= true -Dqfile=vector_decimal_cast.q -Dhive.optimize.index.filter= true and you get this in diffs: Begin query: vector_decimal_cast.q Running: diff -a /Users/ashutosh/workspace/eclipse-workspace/hive/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/vector_decimal_cast.q.out /Users/ashutosh/workspace/eclipse-workspace/hive/itests/qtest/../../ql/src/test/results/clientpositive/tez/vector_decimal_cast.q.out 15d14 < filterExpr: (((cdouble is not null and cint is not null ) and cboolean1 is not null ) and ctimestamp1 is not null ) (type: boolean ) 32a32,41 > -13326.0 528534767 true 1969-12-31 15:59:46.674 -13326.0000000000 528534767.00000000000000 1.00 -13 > -15813.0 528534767 true 1969-12-31 15:59:55.787 -15813.0000000000 528534767.00000000000000 1.00 -4 > -9566.0 528534767 true 1969-12-31 15:59:44.187 -9566.0000000000 528534767.00000000000000 1.00 -16 > 15007.0 528534767 true 1969-12-31 15:59:50.434 15007.0000000000 528534767.00000000000000 1.00 -10 > 7021.0 528534767 true 1969-12-31 16:00:15.007 7021.0000000000 528534767.00000000000000 1.00 15 > 4963.0 528534767 true 1969-12-31 16:00:07.021 4963.0000000000 528534767.00000000000000 1.00 7 > -7824.0 528534767 true 1969-12-31 16:00:04.963 -7824.0000000000 528534767.00000000000000 1.00 5 > -15431.0 528534767 true 1969-12-31 15:59:52.176 -15431.0000000000 528534767.00000000000000 1.00 -8 > -15549.0 528534767 true 1969-12-31 15:59:44.569 -15549.0000000000 528534767.00000000000000 1.00 -15 > 5780.0 528534767 true 1969-12-31 15:59:44.451 5780.0000000000 528534767.00000000000000 1.00 -16

            prasanth_j: Looks like ctimestamp1 column is the culprit here.

            gopalv Gopal Vijayaraghavan added a comment - prasanth_j : Looks like ctimestamp1 column is the culprit here.
            gopalv Gopal Vijayaraghavan added a comment - prasanth_j : review?

            LGTM, +1.

            prasanth_j Prasanth Jayachandran added a comment - LGTM, +1.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12771280/HIVE-12363.1.patch

            ERROR: -1 due to no test(s) being added or modified.

            ERROR: -1 due to 4 failed/errored test(s), 9779 tests executed
            Failed tests:

            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
            org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
            org.apache.hive.jdbc.TestSSL.testSSLVersion
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5989/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5989/console
            Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5989/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 4 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12771280 - PreCommit-HIVE-TRUNK-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12771280/HIVE-12363.1.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 4 failed/errored test(s), 9779 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver org.apache.hive.jdbc.TestSSL.testSSLVersion Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5989/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5989/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5989/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12771280 - PreCommit-HIVE-TRUNK-Build

            Test failures are unrelated.

            Committed to master & branch-1.

            gopalv Gopal Vijayaraghavan added a comment - Test failures are unrelated. Committed to master & branch-1.

            People

              gopalv Gopal Vijayaraghavan
              ashutoshc Ashutosh Chauhan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: