Hive
  1. Hive
  2. HIVE-3315

Propagates filters which are on the join condition transitively

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      explain select src1.key from src src1 join src src2 on src1.key=src2.key and src1.key < 100;

      In this case, filter on join condition src1.key < 100 can be propagated transitively to src2 by src2.key < 100.

      1. HIVE-3315.D4497.7.patch
        117 kB
        Phabricator
      2. HIVE-3315.D4497.5.patch
        123 kB
        Phabricator
      3. HIVE-3315.D4497.6.patch
        123 kB
        Phabricator
      4. HIVE-3315.3.patch.txt
        98 kB
        Navis
      5. HIVE-3315.2.patch.txt
        97 kB
        Navis
      6. HIVE-3315.1.patch.txt
        59 kB
        Navis

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3315 Propagates filters which are on the join condition transitively
        (Navis via namit) (Revision 1391108)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391108
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java
        • /hive/trunk/ql/src/test/queries/clientpositive/join_nullsafe.q
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join11.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join12.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join13.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join14.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join20.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join27.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join28.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/filter_join_breaktask.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join11.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join12.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join13.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join14.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join16.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join20.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join40.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join_nullsafe.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/skewjoin.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3315 Propagates filters which are on the join condition transitively (Navis via namit) (Revision 1391108) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391108 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java /hive/trunk/ql/src/test/queries/clientpositive/join_nullsafe.q /hive/trunk/ql/src/test/results/clientpositive/auto_join11.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join12.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join13.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join14.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join20.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join27.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join28.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out /hive/trunk/ql/src/test/results/clientpositive/filter_join_breaktask.q.out /hive/trunk/ql/src/test/results/clientpositive/join11.q.out /hive/trunk/ql/src/test/results/clientpositive/join12.q.out /hive/trunk/ql/src/test/results/clientpositive/join13.q.out /hive/trunk/ql/src/test/results/clientpositive/join14.q.out /hive/trunk/ql/src/test/results/clientpositive/join16.q.out /hive/trunk/ql/src/test/results/clientpositive/join20.q.out /hive/trunk/ql/src/test/results/clientpositive/join40.q.out /hive/trunk/ql/src/test/results/clientpositive/join_nullsafe.q.out /hive/trunk/ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out /hive/trunk/ql/src/test/results/clientpositive/skewjoin.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1704 (See https://builds.apache.org/job/Hive-trunk-h0.21/1704/)
        HIVE-3315 Propagates filters which are on the join condition transitively
        (Navis via namit) (Revision 1391108)

        Result = FAILURE
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391108
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java
        • /hive/trunk/ql/src/test/queries/clientpositive/join_nullsafe.q
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join11.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join12.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join13.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join14.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join20.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join27.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join28.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/filter_join_breaktask.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join11.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join12.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join13.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join14.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join16.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join20.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join40.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/join_nullsafe.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/skewjoin.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1704 (See https://builds.apache.org/job/Hive-trunk-h0.21/1704/ ) HIVE-3315 Propagates filters which are on the join condition transitively (Navis via namit) (Revision 1391108) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391108 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java /hive/trunk/ql/src/test/queries/clientpositive/join_nullsafe.q /hive/trunk/ql/src/test/results/clientpositive/auto_join11.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join12.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join13.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join14.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join20.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join27.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join28.q.out /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out /hive/trunk/ql/src/test/results/clientpositive/filter_join_breaktask.q.out /hive/trunk/ql/src/test/results/clientpositive/join11.q.out /hive/trunk/ql/src/test/results/clientpositive/join12.q.out /hive/trunk/ql/src/test/results/clientpositive/join13.q.out /hive/trunk/ql/src/test/results/clientpositive/join14.q.out /hive/trunk/ql/src/test/results/clientpositive/join16.q.out /hive/trunk/ql/src/test/results/clientpositive/join20.q.out /hive/trunk/ql/src/test/results/clientpositive/join40.q.out /hive/trunk/ql/src/test/results/clientpositive/join_nullsafe.q.out /hive/trunk/ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out /hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out /hive/trunk/ql/src/test/results/clientpositive/skewjoin.q.out
        Hide
        Namit Jain added a comment -

        Committed. Thanks Navis

        Show
        Namit Jain added a comment - Committed. Thanks Navis
        Hide
        Phabricator added a comment -

        njain has accepted the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        BRANCH
        DPAL-1704

        To: JIRA, njain, navis
        Cc: njain

        Show
        Phabricator added a comment - njain has accepted the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". REVISION DETAIL https://reviews.facebook.net/D4497 BRANCH DPAL-1704 To: JIRA, njain, navis Cc: njain
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".

        looks mostly good - let me review one more time in detail before accepting

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". looks mostly good - let me review one more time in detail before accepting REVISION DETAIL https://reviews.facebook.net/D4497 To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".
        Reviewers: JIRA

        Addressed comments and rebased to trunk

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
        ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java
        ql/src/test/queries/clientpositive/join_nullsafe.q
        ql/src/test/results/clientpositive/auto_join11.q.out
        ql/src/test/results/clientpositive/auto_join12.q.out
        ql/src/test/results/clientpositive/auto_join13.q.out
        ql/src/test/results/clientpositive/auto_join14.q.out
        ql/src/test/results/clientpositive/auto_join16.q.out
        ql/src/test/results/clientpositive/auto_join20.q.out
        ql/src/test/results/clientpositive/auto_join27.q.out
        ql/src/test/results/clientpositive/auto_join28.q.out
        ql/src/test/results/clientpositive/auto_join29.q.out
        ql/src/test/results/clientpositive/bucket_map_join_1.q.out
        ql/src/test/results/clientpositive/bucket_map_join_2.q.out
        ql/src/test/results/clientpositive/filter_join_breaktask.q.out
        ql/src/test/results/clientpositive/join11.q.out
        ql/src/test/results/clientpositive/join12.q.out
        ql/src/test/results/clientpositive/join13.q.out
        ql/src/test/results/clientpositive/join14.q.out
        ql/src/test/results/clientpositive/join16.q.out
        ql/src/test/results/clientpositive/join20.q.out
        ql/src/test/results/clientpositive/join40.q.out
        ql/src/test/results/clientpositive/join_nullsafe.q.out
        ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out
        ql/src/test/results/clientpositive/ppd_gby_join.q.out
        ql/src/test/results/clientpositive/ppd_join.q.out
        ql/src/test/results/clientpositive/ppd_join2.q.out
        ql/src/test/results/clientpositive/ppd_join3.q.out
        ql/src/test/results/clientpositive/regex_col.q.out
        ql/src/test/results/clientpositive/skewjoin.q.out

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - navis updated the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". Reviewers: JIRA Addressed comments and rebased to trunk REVISION DETAIL https://reviews.facebook.net/D4497 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java ql/src/test/queries/clientpositive/join_nullsafe.q ql/src/test/results/clientpositive/auto_join11.q.out ql/src/test/results/clientpositive/auto_join12.q.out ql/src/test/results/clientpositive/auto_join13.q.out ql/src/test/results/clientpositive/auto_join14.q.out ql/src/test/results/clientpositive/auto_join16.q.out ql/src/test/results/clientpositive/auto_join20.q.out ql/src/test/results/clientpositive/auto_join27.q.out ql/src/test/results/clientpositive/auto_join28.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/bucket_map_join_1.q.out ql/src/test/results/clientpositive/bucket_map_join_2.q.out ql/src/test/results/clientpositive/filter_join_breaktask.q.out ql/src/test/results/clientpositive/join11.q.out ql/src/test/results/clientpositive/join12.q.out ql/src/test/results/clientpositive/join13.q.out ql/src/test/results/clientpositive/join14.q.out ql/src/test/results/clientpositive/join16.q.out ql/src/test/results/clientpositive/join20.q.out ql/src/test/results/clientpositive/join40.q.out ql/src/test/results/clientpositive/join_nullsafe.q.out ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out ql/src/test/results/clientpositive/ppd_gby_join.q.out ql/src/test/results/clientpositive/ppd_join.q.out ql/src/test/results/clientpositive/ppd_join2.q.out ql/src/test/results/clientpositive/ppd_join3.q.out ql/src/test/results/clientpositive/regex_col.q.out ql/src/test/results/clientpositive/skewjoin.q.out To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        navis has commented on the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:66 ok.
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:77 ok.
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:183 I think filters on FO is not transitive to opposite alias. I'll add FO case with blank line.
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:38 This method actually replaces expression in sources with expression in targets.
        For example, origin : (a.k+100)a.v, sources = a.k,a.v, targets = b.k+5,b.v+5 than this returns ((b.k+5)+100)(b.v+5)
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:44 Length of sources and targets should be same. In this patch, sources and targets are RS keys for a join operation, which is always same. I'll update comments.
        ql/src/test/results/clientpositive/auto_join29.q.out:3160 Yes, right. That's for HIVE-3381.
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:93 If filter already exists on the alias, filter op need not be created but just be merged with it. And also RS is still needed to insert the filter.

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - navis has commented on the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:66 ok. ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:77 ok. ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:183 I think filters on FO is not transitive to opposite alias. I'll add FO case with blank line. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:38 This method actually replaces expression in sources with expression in targets. For example, origin : (a.k+100) a.v, sources = a.k,a.v, targets = b.k+5,b.v+5 than this returns ((b.k+5)+100) (b.v+5) ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:44 Length of sources and targets should be same. In this patch, sources and targets are RS keys for a join operation, which is always same. I'll update comments. ql/src/test/results/clientpositive/auto_join29.q.out:3160 Yes, right. That's for HIVE-3381 . ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:93 If filter already exists on the alias, filter op need not be created but just be merged with it. And also RS is still needed to insert the filter. REVISION DETAIL https://reviews.facebook.net/D4497 To: JIRA, navis Cc: njain
        Hide
        Namit Jain added a comment -

        Can you refresh ? Some of the changes might not be needed since HIVE-3507 is in

        Show
        Namit Jain added a comment - Can you refresh ? Some of the changes might not be needed since HIVE-3507 is in
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".

        INLINE COMMENTS
        ql/src/test/results/clientpositive/auto_join29.q.out:3160 There is a bug here -

        the new result looks correct if I manually think about the query results.
        But, the results should be the same irrespective of this patch.
        Anyway, outside the scope of this patch.
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:38 the comment is confusing - nothing gets replaced
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:77 nit: can you have ( at the next line - it is confusing
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:66 confusing: can you pass newFilters in the context passed to the rule dispatcher

        same for filterPropagates etc.
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:183 missing full_outer_join
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:44 The size of sources, targets can be diff.

        if targets.size() < index
        return null
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:93 Wont it be simpler to add the filter in JoinTransitive instead of returning this array, and
        then inserting it here ?

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". INLINE COMMENTS ql/src/test/results/clientpositive/auto_join29.q.out:3160 There is a bug here - the new result looks correct if I manually think about the query results. But, the results should be the same irrespective of this patch. Anyway, outside the scope of this patch. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:38 the comment is confusing - nothing gets replaced ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:77 nit: can you have ( at the next line - it is confusing ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:66 confusing: can you pass newFilters in the context passed to the rule dispatcher same for filterPropagates etc. ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:183 missing full_outer_join ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java:44 The size of sources, targets can be diff. if targets.size() < index return null ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java:93 Wont it be simpler to add the filter in JoinTransitive instead of returning this array, and then inserting it here ? REVISION DETAIL https://reviews.facebook.net/D4497 To: JIRA, navis Cc: njain
        Hide
        Namit Jain added a comment -

        comments on phabricator

        Show
        Namit Jain added a comment - comments on phabricator
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".
        Reviewers: JIRA

        1. Simplified code
        2. Support propagating filter on outer join condition

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
        ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java
        ql/src/test/queries/clientpositive/join_nullsafe.q
        ql/src/test/results/clientpositive/auto_join11.q.out
        ql/src/test/results/clientpositive/auto_join12.q.out
        ql/src/test/results/clientpositive/auto_join13.q.out
        ql/src/test/results/clientpositive/auto_join14.q.out
        ql/src/test/results/clientpositive/auto_join16.q.out
        ql/src/test/results/clientpositive/auto_join20.q.out
        ql/src/test/results/clientpositive/auto_join27.q.out
        ql/src/test/results/clientpositive/auto_join28.q.out
        ql/src/test/results/clientpositive/auto_join29.q.out
        ql/src/test/results/clientpositive/bucket_map_join_1.q.out
        ql/src/test/results/clientpositive/bucket_map_join_2.q.out
        ql/src/test/results/clientpositive/filter_join_breaktask.q.out
        ql/src/test/results/clientpositive/join11.q.out
        ql/src/test/results/clientpositive/join12.q.out
        ql/src/test/results/clientpositive/join13.q.out
        ql/src/test/results/clientpositive/join14.q.out
        ql/src/test/results/clientpositive/join16.q.out
        ql/src/test/results/clientpositive/join20.q.out
        ql/src/test/results/clientpositive/join40.q.out
        ql/src/test/results/clientpositive/join_nullsafe.q.out
        ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out
        ql/src/test/results/clientpositive/ppd_gby_join.q.out
        ql/src/test/results/clientpositive/ppd_join.q.out
        ql/src/test/results/clientpositive/ppd_join2.q.out
        ql/src/test/results/clientpositive/ppd_join3.q.out
        ql/src/test/results/clientpositive/regex_col.q.out
        ql/src/test/results/clientpositive/skewjoin.q.out
        ql/src/test/results/clientpositive/skewjoinopt1.q.out
        ql/src/test/results/clientpositive/skewjoinopt10.q.out
        ql/src/test/results/clientpositive/skewjoinopt15.q.out
        ql/src/test/results/clientpositive/skewjoinopt16.q.out
        ql/src/test/results/clientpositive/skewjoinopt17.q.out
        ql/src/test/results/clientpositive/skewjoinopt19.q.out
        ql/src/test/results/clientpositive/skewjoinopt2.q.out
        ql/src/test/results/clientpositive/skewjoinopt20.q.out
        ql/src/test/results/clientpositive/skewjoinopt3.q.out
        ql/src/test/results/clientpositive/skewjoinopt4.q.out
        ql/src/test/results/clientpositive/skewjoinopt5.q.out

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - navis updated the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". Reviewers: JIRA 1. Simplified code 2. Support propagating filter on outer join condition REVISION DETAIL https://reviews.facebook.net/D4497 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java ql/src/test/queries/clientpositive/join_nullsafe.q ql/src/test/results/clientpositive/auto_join11.q.out ql/src/test/results/clientpositive/auto_join12.q.out ql/src/test/results/clientpositive/auto_join13.q.out ql/src/test/results/clientpositive/auto_join14.q.out ql/src/test/results/clientpositive/auto_join16.q.out ql/src/test/results/clientpositive/auto_join20.q.out ql/src/test/results/clientpositive/auto_join27.q.out ql/src/test/results/clientpositive/auto_join28.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/bucket_map_join_1.q.out ql/src/test/results/clientpositive/bucket_map_join_2.q.out ql/src/test/results/clientpositive/filter_join_breaktask.q.out ql/src/test/results/clientpositive/join11.q.out ql/src/test/results/clientpositive/join12.q.out ql/src/test/results/clientpositive/join13.q.out ql/src/test/results/clientpositive/join14.q.out ql/src/test/results/clientpositive/join16.q.out ql/src/test/results/clientpositive/join20.q.out ql/src/test/results/clientpositive/join40.q.out ql/src/test/results/clientpositive/join_nullsafe.q.out ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out ql/src/test/results/clientpositive/ppd_gby_join.q.out ql/src/test/results/clientpositive/ppd_join.q.out ql/src/test/results/clientpositive/ppd_join2.q.out ql/src/test/results/clientpositive/ppd_join3.q.out ql/src/test/results/clientpositive/regex_col.q.out ql/src/test/results/clientpositive/skewjoin.q.out ql/src/test/results/clientpositive/skewjoinopt1.q.out ql/src/test/results/clientpositive/skewjoinopt10.q.out ql/src/test/results/clientpositive/skewjoinopt15.q.out ql/src/test/results/clientpositive/skewjoinopt16.q.out ql/src/test/results/clientpositive/skewjoinopt17.q.out ql/src/test/results/clientpositive/skewjoinopt19.q.out ql/src/test/results/clientpositive/skewjoinopt2.q.out ql/src/test/results/clientpositive/skewjoinopt20.q.out ql/src/test/results/clientpositive/skewjoinopt3.q.out ql/src/test/results/clientpositive/skewjoinopt4.q.out ql/src/test/results/clientpositive/skewjoinopt5.q.out To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".
        Reviewers: JIRA

        Rebased to trunk

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
        ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
        ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java
        ql/src/test/queries/clientpositive/join_nullsafe.q
        ql/src/test/results/clientpositive/auto_join11.q.out
        ql/src/test/results/clientpositive/auto_join12.q.out
        ql/src/test/results/clientpositive/auto_join13.q.out
        ql/src/test/results/clientpositive/auto_join14.q.out
        ql/src/test/results/clientpositive/auto_join16.q.out
        ql/src/test/results/clientpositive/auto_join20.q.out
        ql/src/test/results/clientpositive/auto_join27.q.out
        ql/src/test/results/clientpositive/auto_join28.q.out
        ql/src/test/results/clientpositive/auto_join29.q.out
        ql/src/test/results/clientpositive/bucket_map_join_1.q.out
        ql/src/test/results/clientpositive/bucket_map_join_2.q.out
        ql/src/test/results/clientpositive/filter_join_breaktask.q.out
        ql/src/test/results/clientpositive/join11.q.out
        ql/src/test/results/clientpositive/join12.q.out
        ql/src/test/results/clientpositive/join13.q.out
        ql/src/test/results/clientpositive/join14.q.out
        ql/src/test/results/clientpositive/join16.q.out
        ql/src/test/results/clientpositive/join20.q.out
        ql/src/test/results/clientpositive/join40.q.out
        ql/src/test/results/clientpositive/join_nullsafe.q.out
        ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out
        ql/src/test/results/clientpositive/ppd_gby_join.q.out
        ql/src/test/results/clientpositive/ppd_join.q.out
        ql/src/test/results/clientpositive/ppd_join2.q.out
        ql/src/test/results/clientpositive/ppd_join3.q.out
        ql/src/test/results/clientpositive/regex_col.q.out
        ql/src/test/results/clientpositive/skewjoin.q.out
        ql/src/test/results/clientpositive/skewjoinopt1.q.out
        ql/src/test/results/clientpositive/skewjoinopt10.q.out
        ql/src/test/results/clientpositive/skewjoinopt15.q.out
        ql/src/test/results/clientpositive/skewjoinopt16.q.out
        ql/src/test/results/clientpositive/skewjoinopt17.q.out
        ql/src/test/results/clientpositive/skewjoinopt19.q.out
        ql/src/test/results/clientpositive/skewjoinopt2.q.out
        ql/src/test/results/clientpositive/skewjoinopt20.q.out
        ql/src/test/results/clientpositive/skewjoinopt3.q.out
        ql/src/test/results/clientpositive/skewjoinopt4.q.out
        ql/src/test/results/clientpositive/skewjoinopt5.q.out

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - navis updated the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". Reviewers: JIRA Rebased to trunk REVISION DETAIL https://reviews.facebook.net/D4497 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicateTransitivePropagate.java ql/src/test/queries/clientpositive/join_nullsafe.q ql/src/test/results/clientpositive/auto_join11.q.out ql/src/test/results/clientpositive/auto_join12.q.out ql/src/test/results/clientpositive/auto_join13.q.out ql/src/test/results/clientpositive/auto_join14.q.out ql/src/test/results/clientpositive/auto_join16.q.out ql/src/test/results/clientpositive/auto_join20.q.out ql/src/test/results/clientpositive/auto_join27.q.out ql/src/test/results/clientpositive/auto_join28.q.out ql/src/test/results/clientpositive/auto_join29.q.out ql/src/test/results/clientpositive/bucket_map_join_1.q.out ql/src/test/results/clientpositive/bucket_map_join_2.q.out ql/src/test/results/clientpositive/filter_join_breaktask.q.out ql/src/test/results/clientpositive/join11.q.out ql/src/test/results/clientpositive/join12.q.out ql/src/test/results/clientpositive/join13.q.out ql/src/test/results/clientpositive/join14.q.out ql/src/test/results/clientpositive/join16.q.out ql/src/test/results/clientpositive/join20.q.out ql/src/test/results/clientpositive/join40.q.out ql/src/test/results/clientpositive/join_nullsafe.q.out ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out ql/src/test/results/clientpositive/ppd_gby_join.q.out ql/src/test/results/clientpositive/ppd_join.q.out ql/src/test/results/clientpositive/ppd_join2.q.out ql/src/test/results/clientpositive/ppd_join3.q.out ql/src/test/results/clientpositive/regex_col.q.out ql/src/test/results/clientpositive/skewjoin.q.out ql/src/test/results/clientpositive/skewjoinopt1.q.out ql/src/test/results/clientpositive/skewjoinopt10.q.out ql/src/test/results/clientpositive/skewjoinopt15.q.out ql/src/test/results/clientpositive/skewjoinopt16.q.out ql/src/test/results/clientpositive/skewjoinopt17.q.out ql/src/test/results/clientpositive/skewjoinopt19.q.out ql/src/test/results/clientpositive/skewjoinopt2.q.out ql/src/test/results/clientpositive/skewjoinopt20.q.out ql/src/test/results/clientpositive/skewjoinopt3.q.out ql/src/test/results/clientpositive/skewjoinopt4.q.out ql/src/test/results/clientpositive/skewjoinopt5.q.out To: JIRA, navis Cc: njain
        Hide
        Phabricator added a comment -

        navis has commented on the revision "HIVE-3315 [jira] Propagate filers on inner join condition transitively".

        This is a little old issue and I've forgot what was this for. I've taken sometime for remind.

        I agree on what you said generally (too complex) and re-implemented as you said, except filter propagation of outer joins. For inner/semi join, filters can be propagated freely(L to R and R to L). But for left-outer join, could it be propagated in L to R only? (R to L for RO)

        I'm still cannot sure of this. I'll update after full test.

        REVISION DETAIL
        https://reviews.facebook.net/D4497

        To: JIRA, navis
        Cc: njain

        Show
        Phabricator added a comment - navis has commented on the revision " HIVE-3315 [jira] Propagate filers on inner join condition transitively". This is a little old issue and I've forgot what was this for. I've taken sometime for remind. I agree on what you said generally (too complex) and re-implemented as you said, except filter propagation of outer joins. For inner/semi join, filters can be propagated freely(L to R and R to L). But for left-outer join, could it be propagated in L to R only? (R to L for RO) I'm still cannot sure of this. I'll update after full test. REVISION DETAIL https://reviews.facebook.net/D4497 To: JIRA, navis Cc: njain
        Hide
        Namit Jain added a comment -

        comments on phabricator

        Show
        Namit Jain added a comment - comments on phabricator
        Hide
        Navis added a comment -

        Fixed bug

        Show
        Navis added a comment - Fixed bug
        Hide
        Navis added a comment -

        I've misunderstood join conditions. Will be updated soon.

        Show
        Navis added a comment - I've misunderstood join conditions. Will be updated soon.
        Hide
        Navis added a comment -

        Addressed comments

        Show
        Navis added a comment - Addressed comments
        Hide
        Namit Jain added a comment -

        comments on phabricator

        Show
        Namit Jain added a comment - comments on phabricator
        Hide
        Navis added a comment -
        Show
        Navis added a comment - https://reviews.facebook.net/D4497

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development