Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9278

Cached expression feature broken in one case

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 1.0.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      Different query result depending on whether hive.cache.expr.evaluation is true or false. When true, no query results are produced (this is wrong).

      The q file:

      set hive.cache.expr.evaluation=true;
      
      CREATE TABLE cache_expr_repro (date_str STRING);
      LOAD DATA LOCAL INPATH '../../data/files/cache_expr_repro.txt' INTO TABLE cache_expr_repro;
      
      SELECT MONTH(date_str) AS `mon`, CAST((MONTH(date_str) - 1) / 3 + 1 AS int) AS `quarter`,   YEAR(date_str) AS `year` FROM cache_expr_repro WHERE ((CAST((MONTH(date_str) - 1) / 3 + 1 AS int) = 1) AND (YEAR(date_str) = 2015)) GROUP BY MONTH(date_str), CAST((MONTH(date_str) - 1) / 3 + 1 AS int),   YEAR(date_str) ;
      

      cache_expr_repro.txt

      2015-01-01 00:00:00
      2015-02-01 00:00:00
      2015-01-01 00:00:00
      2015-02-01 00:00:00
      2015-01-01 00:00:00
      2015-01-01 00:00:00
      2015-02-01 00:00:00
      2015-02-01 00:00:00
      2015-01-01 00:00:00
      2015-01-01 00:00:00
      

        Issue Links

          Activity

          Hide
          navis Navis added a comment -

          My bad, again.

          Show
          navis Navis added a comment - My bad, again.
          Hide
          hiveqa Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12690495/HIVE-9278.1.patch.txt

          SUCCESS: +1 6724 tests passed

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2279/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2279/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2279/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12690495 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12690495/HIVE-9278.1.patch.txt SUCCESS: +1 6724 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2279/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2279/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2279/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12690495 - PreCommit-HIVE-TRUNK-Build
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          I don't follow the testcase. Can't understand how expression cache will be used in your .q test, since no expression is repeated in that test. Navis Would you mind adding Matt's test in your test case, so that we have more confidence in the fix.

          Show
          ashutoshc Ashutosh Chauhan added a comment - I don't follow the testcase. Can't understand how expression cache will be used in your .q test, since no expression is repeated in that test. Navis Would you mind adding Matt's test in your test case, so that we have more confidence in the fix.
          Hide
          navis Navis added a comment -

          Ashutosh Chauhan In caching, identity was checked by comparing toString(). But for UDFs (not GenericUDF), these return always same class name (org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge) making them shared between different expressions. In the testcase, "length(key)" and "reverse(key)" are both UDFs, resulting length(key)=reverse(key) always. Now it's checked correctly with ExprNodeDesc itself (with isSame() method).

          Show
          navis Navis added a comment - Ashutosh Chauhan In caching, identity was checked by comparing toString(). But for UDFs (not GenericUDF), these return always same class name (org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge) making them shared between different expressions. In the testcase, "length(key)" and "reverse(key)" are both UDFs, resulting length(key)=reverse(key) always. Now it's checked correctly with ExprNodeDesc itself (with isSame() method).
          Hide
          navis Navis added a comment -

          Should be included in hive-0.14.1

          Show
          navis Navis added a comment - Should be included in hive-0.14.1
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Aah.. I see. +1

          Show
          ashutoshc Ashutosh Chauhan added a comment - Aah.. I see. +1
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Committed to trunk.
          Vikram Dixit K This is a correctness issue. Recommended to be included in 0.14.1

          Show
          ashutoshc Ashutosh Chauhan added a comment - Committed to trunk. Vikram Dixit K This is a correctness issue. Recommended to be included in 0.14.1
          Hide
          vikram.dixit Vikram Dixit K added a comment -

          +1 for 0.14

          Show
          vikram.dixit Vikram Dixit K added a comment - +1 for 0.14
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Committed to 0.14 as well.

          Show
          ashutoshc Ashutosh Chauhan added a comment - Committed to 0.14 as well.
          Hide
          thejas Thejas M Nair added a comment -

          Updating release version for jiras resolved in 1.0.0 .

          Show
          thejas Thejas M Nair added a comment - Updating release version for jiras resolved in 1.0.0 .
          Hide
          thejas Thejas M Nair added a comment -

          This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          Show
          thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

            People

            • Assignee:
              navis Navis
              Reporter:
              mmccline Matt McCline
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development