Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.1.0
    • CBO
    • None

    Description

      Follow-up of HIVE-13068.

      Extension to folding expressions for NOT.

      Currently, simplification is performed only if NOT is applied on a simple operation (e.g. IS NOT NULL, =, <>, etc.). We should take advantage of NOT distributivity when it is applied on OR/AND operations to try to simplify predicates further.

      Ex. ql/src/test/results/clientpositive/folder_predicate.q.out

      explain
      SELECT * FROM predicate_fold_tb WHERE not(value IS NOT NULL AND value = 3)
      

      Plan:

      STAGE DEPENDENCIES:
        Stage-1 is a root stage
        Stage-0 depends on stages: Stage-1
      
      STAGE PLANS:
        Stage: Stage-1
          Map Reduce
            Map Operator Tree:
                TableScan
                  alias: predicate_fold_tb
                  Statistics: Num rows: 6 Data size: 7 Basic stats: COMPLETE Column stats: NONE
                  Filter Operator
                    predicate: (not (value is not null and (value = 3))) (type: boolean)
                    Statistics: Num rows: 3 Data size: 3 Basic stats: COMPLETE Column stats: NONE
                    Select Operator
                      expressions: value (type: int)
                      outputColumnNames: _col0
                      Statistics: Num rows: 3 Data size: 3 Basic stats: COMPLETE Column stats: NONE
                      File Output Operator
                        compressed: false
                        Statistics: Num rows: 3 Data size: 3 Basic stats: COMPLETE Column stats: NONE
                        table:
                            input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                            output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                            serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
      
        Stage: Stage-0
          Fetch Operator
            limit: -1
            Processor Tree:
              ListSink
      

      Attachments

        1. HIVE-13806.patch
          10 kB
          jcamachorodriguez
        2. HIVE-13806.01.patch
          19 kB
          jcamachorodriguez

        Issue Links

          Activity

            +1 will likely require updates to other golden files as well.

            ashutoshc Ashutosh Chauhan added a comment - +1 will likely require updates to other golden files as well.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12807475/HIVE-13806.patch

            ERROR: -1 due to no test(s) being added or modified.

            ERROR: -1 due to 10 failed/errored test(s), 10197 tests executed
            Failed tests:

            TestJdbcWithMiniHA - did not produce a TEST-*.xml file
            TestJdbcWithMiniMr - did not produce a TEST-*.xml file
            TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner3
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_offcbo
            org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
            org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
            

            Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/506/testReport
            Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/506/console
            Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-506/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 10 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12807475 - PreCommit-HIVE-MASTER-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12807475/HIVE-13806.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 10 failed/errored test(s), 10197 tests executed Failed tests: TestJdbcWithMiniHA - did not produce a TEST-*.xml file TestJdbcWithMiniMr - did not produce a TEST-*.xml file TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rand_partitionpruner3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_offcbo org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/506/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/506/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-506/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12807475 - PreCommit-HIVE-MASTER-Build
            jcamacho Jesús Camacho Rodríguez added a comment - - edited

            Pushed to master, thanks for reviewing ashutoshc!

            jcamacho Jesús Camacho Rodríguez added a comment - - edited Pushed to master, thanks for reviewing ashutoshc !
            jcamacho Jesús Camacho Rodríguez added a comment - - edited

            Pushed to branch-2.1.

            jcamacho Jesús Camacho Rodríguez added a comment - - edited Pushed to branch-2.1.

            People

              jcamacho Jesús Camacho Rodríguez
              jcamacho Jesús Camacho Rodríguez
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: