Hive
  1. Hive
  2. HIVE-7304

Transitive Predicate Propagation doesn't happen properly after HIVE-7159

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      The reason for the regression is that PredicateTransitivePropagate looks at the FilterOperator below the ReduceSink.
      SemanticAnalyzer::genNotNullFilterForJoinSourcePlan was stacking another FilterOp for the not null check, so only that predicate was being applied transitively by PredicateTransitivePropagate.

      1. HIVE-7304.1.patch
        52 kB
        Harish Butani
      2. HIVE-7304.2.patch
        97 kB
        Harish Butani

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        4h 16m 1 Harish Butani 27/Jun/14 22:33
        Open Open Patch Available Patch Available
        1h 52m 2 Harish Butani 27/Jun/14 22:33
        Patch Available Patch Available Resolved Resolved
        3d 6h 29m 1 Ashutosh Chauhan 01/Jul/14 05:02
        Resolved Resolved Closed Closed
        135d 15h 39m 1 Thejas M Nair 13/Nov/14 19:42
        Thejas M Nair made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
        Hide
        Gopal V added a comment -

        Ashutosh Chauhan: I agree that the SARGs and filters shouldn't have anything to do with each other, except that one is derived from the other.

        SARGS currently use YES, NO, YES_NULL, NO_NULL & NULL as potential evaluation return types.

        While filters are limited to YES & NO (or TRUE/FALSE).

        Show
        Gopal V added a comment - Ashutosh Chauhan : I agree that the SARGs and filters shouldn't have anything to do with each other, except that one is derived from the other. SARGS currently use YES, NO, YES_NULL, NO_NULL & NULL as potential evaluation return types. While filters are limited to YES & NO (or TRUE/FALSE).
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.14.0 [ 12326450 ]
        Resolution Fixed [ 1 ]
        Hide
        Ashutosh Chauhan added a comment -

        I see. In those cases, storage handler should be able to make that kind of inference itself. Storage handler need not to be told from planner that it can extend filters (as evident in this example) itself.

        Anyways, committed this one to trunk. Thanks, Harish!

        Show
        Ashutosh Chauhan added a comment - I see. In those cases, storage handler should be able to make that kind of inference itself. Storage handler need not to be told from planner that it can extend filters (as evident in this example) itself. Anyways, committed this one to trunk. Thanks, Harish!
        Hide
        Harish Butani added a comment -

        Ashutosh Chauhan yes i would prefer committing this one.
        Also Gopal V pointed out that in the above e.g. 'key is not null' is useful(performance wise) because the Storage Handler predicate evaluation of 'key < 100' doesn't filter out null valued columns.

        Show
        Harish Butani added a comment - Ashutosh Chauhan yes i would prefer committing this one. Also Gopal V pointed out that in the above e.g. 'key is not null' is useful(performance wise) because the Storage Handler predicate evaluation of 'key < 100' doesn't filter out null valued columns.
        Hide
        Ashutosh Chauhan added a comment -

        Harish Butani Shall we commit this one and address above concern in a follow-up jira or do you want to take that up in this one only. I am fine either ways.

        Show
        Ashutosh Chauhan added a comment - Harish Butani Shall we commit this one and address above concern in a follow-up jira or do you want to take that up in this one only. I am fine either ways.
        Hide
        Ashutosh Chauhan added a comment -

        Is not null check should be added only if there is no other filter predicate already being present. eg, we are now getting filter predicates as follows where is not null is redundant.

        ((key < 100) and key is not null)
        

        Perhaps, could be taken up in a follow-up.

        Show
        Ashutosh Chauhan added a comment - Is not null check should be added only if there is no other filter predicate already being present. eg, we are now getting filter predicates as follows where is not null is redundant. ((key < 100) and key is not null ) Perhaps, could be taken up in a follow-up.
        Hide
        Navis added a comment -

        LGTM. +1

        Show
        Navis added a comment - LGTM. +1
        Hide
        Harish Butani added a comment -

        Navis can you please review this patch

        Show
        Harish Butani added a comment - Navis can you please review this patch
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12652901/HIVE-7304.2.patch

        ERROR: -1 due to 1 failed/errored test(s), 5655 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-621/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12652901

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652901/HIVE-7304.2.patch ERROR: -1 due to 1 failed/errored test(s), 5655 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-621/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12652901
        Harish Butani made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Harish Butani made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Harish Butani made changes -
        Attachment HIVE-7304.2.patch [ 12652901 ]
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12652864/HIVE-7304.1.patch

        ERROR: -1 due to 13 failed/errored test(s), 5655 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby_join
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-619/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 13 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12652864

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652864/HIVE-7304.1.patch ERROR: -1 due to 13 failed/errored test(s), 5655 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-619/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed This message is automatically generated. ATTACHMENT ID: 12652864
        Harish Butani made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Harish Butani made changes -
        Field Original Value New Value
        Attachment HIVE-7304.1.patch [ 12652864 ]
        Harish Butani created issue -

          People

          • Assignee:
            Harish Butani
            Reporter:
            Harish Butani
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development