Details

      Description

      With the dataset attached by Tony, running:

      select max(ctinyint), max(csmallint), max(cint), max(cbigint), max(cfloat), max(cdouble) from alltypes_orc;

      this is the result in rowmode:
      OK
      62 16379 1073737829 2146661184 62.0 16376.0

      This is the result in vector mode:
      OK
      1 16379 1071926788 -106115134 62.0 16361.0

      1. HIVE-4931.1.patch
        18 kB
        Remus Rusanu

        Activity

        Hide
        Remus Rusanu added a comment -

        The agg templates were incorrectly checking for isNull[0] in addition of hasNulls for repeating values. I also discovered that the MIN/MAX aggregate was using hard-coded comparison instead of templatized <Operand> for this case.

        These are the results with the patch applied:

        select max(ctinyint), max(csmallint), max(cint), max(cbigint), max(cfloat), max(cdouble) from alltypes_orc;
        rowmode
        62 16379 1073737829 2146661184 62.0 16376.0
        vect:
        62 16379 1073737829 2146661184 62.0 16376.0

        select min(ctinyint), min(csmallint), min(cint), min(cbigint), min(cfloat), min(cdouble) from alltypes_orc;
        rowmode:
        -64 -16381 -1073517051 -2147049514 -64.0 -16376.0
        vect:
        -64 -16381 -1073517051 -2147049514 -64.0 -16376.0

        select avg(ctinyint), avg(csmallint), avg(cint), avg(cbigint), avg(cfloat), avg(cdouble) from alltypes_orc;
        rowmode:
        2.522511719175842 -415.2897318508829 -1.0994357654011336E8 1.9003309384238064E7 -2.2073250490516676 -2877.6021364726403
        vect:
        2.522511719175842 -415.2897318508829 -1.0994357654011336E8 1.9003309384238064E7 -2.2073250490516676 -2877.6021364726403

        select stddev(ctinyint), stddev(csmallint), stddev(cint), stddev(cbigint), stddev(cfloat), stddev(cdouble) from alltypes_orc;
        rowmode:
        34.52579284952266 8956.673250179136 6.765972128417507E8 1.0284485805202962E9 38.5241575060334 10107.644590369582
        vect:
        34.52579284952266 8956.673250179136 6.765972128417507E8 1.0284485805202962E9 38.5241575060334 10107.644590369582

        Show
        Remus Rusanu added a comment - The agg templates were incorrectly checking for isNull [0] in addition of hasNulls for repeating values. I also discovered that the MIN/MAX aggregate was using hard-coded comparison instead of templatized <Operand> for this case. These are the results with the patch applied: select max(ctinyint), max(csmallint), max(cint), max(cbigint), max(cfloat), max(cdouble) from alltypes_orc; rowmode 62 16379 1073737829 2146661184 62.0 16376.0 vect: 62 16379 1073737829 2146661184 62.0 16376.0 select min(ctinyint), min(csmallint), min(cint), min(cbigint), min(cfloat), min(cdouble) from alltypes_orc; rowmode: -64 -16381 -1073517051 -2147049514 -64.0 -16376.0 vect: -64 -16381 -1073517051 -2147049514 -64.0 -16376.0 select avg(ctinyint), avg(csmallint), avg(cint), avg(cbigint), avg(cfloat), avg(cdouble) from alltypes_orc; rowmode: 2.522511719175842 -415.2897318508829 -1.0994357654011336E8 1.9003309384238064E7 -2.2073250490516676 -2877.6021364726403 vect: 2.522511719175842 -415.2897318508829 -1.0994357654011336E8 1.9003309384238064E7 -2.2073250490516676 -2877.6021364726403 select stddev(ctinyint), stddev(csmallint), stddev(cint), stddev(cbigint), stddev(cfloat), stddev(cdouble) from alltypes_orc; rowmode: 34.52579284952266 8956.673250179136 6.765972128417507E8 1.0284485805202962E9 38.5241575060334 10107.644590369582 vect: 34.52579284952266 8956.673250179136 6.765972128417507E8 1.0284485805202962E9 38.5241575060334 10107.644590369582
        Show
        Remus Rusanu added a comment - https://reviews.apache.org/r/12939/
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12594134/HIVE-4931.1.patch

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/185/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/185/console

        Messages:

        Executing org.apache.hive.ptest.execution.CleanupPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-185/source-prep.txt
        + mkdir -p maven ivy
        + [[ svn = \s\v\n ]]
        + [[ -n '' ]]
        + [[ -d apache-svn-trunk-source ]]
        + [[ ! -d apache-svn-trunk-source/.svn ]]
        + [[ ! -d apache-svn-trunk-source ]]
        + cd apache-svn-trunk-source
        + svn revert -R .
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java'
        ++ egrep -v '^X|^Performing status on external'
        ++ awk '{print $2}'
        ++ svn status --no-ignore
        + rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen
        + svn update
        
        Fetching external item into 'hcatalog/src/test/e2e/harness'
        External at revision 1507115.
        
        At revision 1507115.
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0 to p2
        + exit 1
        '
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594134/HIVE-4931.1.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/185/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/185/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-185/source-prep.txt + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1507115. At revision 1507115. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0 to p2 + exit 1 ' This message is automatically generated.
        Hide
        Ashutosh Chauhan added a comment -

        Committed to branch. Thanks, Remus!

        Show
        Ashutosh Chauhan added a comment - Committed to branch. Thanks, Remus!
        Hide
        Edward Capriolo added a comment - - edited

        Why no tests? What is to prevent a regression?

        Show
        Edward Capriolo added a comment - - edited Why no tests? What is to prevent a regression?
        Hide
        Edward Capriolo added a comment -
        -@Override
        +    @Override
        

        And ^ means the last commit on this file went in without check-style.

        Show
        Edward Capriolo added a comment - -@Override + @Override And ^ means the last commit on this file went in without check-style.
        Hide
        Tony Murphy added a comment -

        Hey Edward. I'm working with Remus and others to stabilize vectorization, and this bug was found as a result of that stabilization. I've just uploaded a new patch for HIVE-4794 which contains the tests we've been using to stabilize, and which give us regression coverage for this issue.

        Show
        Tony Murphy added a comment - Hey Edward. I'm working with Remus and others to stabilize vectorization, and this bug was found as a result of that stabilization. I've just uploaded a new patch for HIVE-4794 which contains the tests we've been using to stabilize, and which give us regression coverage for this issue.
        Hide
        Edward Capriolo added a comment -

        Test patch in a follow up. Marking as closed.

        Show
        Edward Capriolo added a comment - Test patch in a follow up. Marking as closed.

          People

          • Assignee:
            Remus Rusanu
            Reporter:
            Remus Rusanu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development