Hive
  1. Hive
  2. HIVE-7304

Transitive Predicate Propagation doesn't happen properly after HIVE-7159

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      The reason for the regression is that PredicateTransitivePropagate looks at the FilterOperator below the ReduceSink.
      SemanticAnalyzer::genNotNullFilterForJoinSourcePlan was stacking another FilterOp for the not null check, so only that predicate was being applied transitively by PredicateTransitivePropagate.

      1. HIVE-7304.1.patch
        52 kB
        Harish Butani
      2. HIVE-7304.2.patch
        97 kB
        Harish Butani

        Activity

        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12652864/HIVE-7304.1.patch

        ERROR: -1 due to 13 failed/errored test(s), 5655 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby_join
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-619/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 13 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12652864

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652864/HIVE-7304.1.patch ERROR: -1 due to 13 failed/errored test(s), 5655 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regex_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/619/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-619/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed This message is automatically generated. ATTACHMENT ID: 12652864
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12652901/HIVE-7304.2.patch

        ERROR: -1 due to 1 failed/errored test(s), 5655 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-621/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12652901

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652901/HIVE-7304.2.patch ERROR: -1 due to 1 failed/errored test(s), 5655 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/621/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-621/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12652901
        Hide
        Harish Butani added a comment -

        Navis can you please review this patch

        Show
        Harish Butani added a comment - Navis can you please review this patch
        Hide
        Navis added a comment -

        LGTM. +1

        Show
        Navis added a comment - LGTM. +1
        Hide
        Ashutosh Chauhan added a comment -

        Is not null check should be added only if there is no other filter predicate already being present. eg, we are now getting filter predicates as follows where is not null is redundant.

        ((key < 100) and key is not null)
        

        Perhaps, could be taken up in a follow-up.

        Show
        Ashutosh Chauhan added a comment - Is not null check should be added only if there is no other filter predicate already being present. eg, we are now getting filter predicates as follows where is not null is redundant. ((key < 100) and key is not null ) Perhaps, could be taken up in a follow-up.
        Hide
        Ashutosh Chauhan added a comment -

        Harish Butani Shall we commit this one and address above concern in a follow-up jira or do you want to take that up in this one only. I am fine either ways.

        Show
        Ashutosh Chauhan added a comment - Harish Butani Shall we commit this one and address above concern in a follow-up jira or do you want to take that up in this one only. I am fine either ways.
        Hide
        Harish Butani added a comment -

        Ashutosh Chauhan yes i would prefer committing this one.
        Also Gopal V pointed out that in the above e.g. 'key is not null' is useful(performance wise) because the Storage Handler predicate evaluation of 'key < 100' doesn't filter out null valued columns.

        Show
        Harish Butani added a comment - Ashutosh Chauhan yes i would prefer committing this one. Also Gopal V pointed out that in the above e.g. 'key is not null' is useful(performance wise) because the Storage Handler predicate evaluation of 'key < 100' doesn't filter out null valued columns.
        Hide
        Ashutosh Chauhan added a comment -

        I see. In those cases, storage handler should be able to make that kind of inference itself. Storage handler need not to be told from planner that it can extend filters (as evident in this example) itself.

        Anyways, committed this one to trunk. Thanks, Harish!

        Show
        Ashutosh Chauhan added a comment - I see. In those cases, storage handler should be able to make that kind of inference itself. Storage handler need not to be told from planner that it can extend filters (as evident in this example) itself. Anyways, committed this one to trunk. Thanks, Harish!
        Hide
        Gopal V added a comment -

        Ashutosh Chauhan: I agree that the SARGs and filters shouldn't have anything to do with each other, except that one is derived from the other.

        SARGS currently use YES, NO, YES_NULL, NO_NULL & NULL as potential evaluation return types.

        While filters are limited to YES & NO (or TRUE/FALSE).

        Show
        Gopal V added a comment - Ashutosh Chauhan : I agree that the SARGs and filters shouldn't have anything to do with each other, except that one is derived from the other. SARGS currently use YES, NO, YES_NULL, NO_NULL & NULL as potential evaluation return types. While filters are limited to YES & NO (or TRUE/FALSE).
        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

          People

          • Assignee:
            Harish Butani
            Reporter:
            Harish Butani
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development