Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-448

FilterIntoJoinRule creates filters containing invalid RexInputRef

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.2-incubating
    • Fix Version/s: 1.0.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      Filter should be allowed to be constructed even when types are INTEGER && INTEGER NOT NULL respectively. Currently, it fails with :

       java.lang.AssertionError: type mismatch:
      type1:
      INTEGER NOT NULL
      type2:
      INTEGER
      	at org.eigenbase.relopt.RelOptUtil.eq(RelOptUtil.java:1566)
      	at org.eigenbase.rex.RexProgramBuilder$RegisterInputShuttle.visitInputRef(RexProgramBuilder.java:899)
      	at org.eigenbase.rex.RexProgramBuilder$RegisterInputShuttle.visitInputRef(RexProgramBuilder.java:878)
      	at org.eigenbase.rex.RexInputRef.accept(RexInputRef.java:102)
      	at org.eigenbase.rex.RexShuttle.visitList(RexShuttle.java:129)
      	at org.eigenbase.rex.RexShuttle.visitCall(RexShuttle.java:78)
      	at org.eigenbase.rex.RexProgramBuilder$RegisterShuttle.visitCall(RexProgramBuilder.java:843)
      	at org.eigenbase.rex.RexProgramBuilder$RegisterShuttle.visitCall(RexProgramBuilder.java:841)
      	at org.eigenbase.rex.RexCall.accept(RexCall.java:105)
      	at org.eigenbase.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:272)
      	at org.eigenbase.rex.RexProgramBuilder.addCondition(RexProgramBuilder.java:247)
      	at org.eigenbase.relopt.RelOptUtil.pushFilterPastProject(RelOptUtil.java:2367)
      	at org.eigenbase.rel.rules.PushFilterPastProjectRule.onMatch(PushFilterPastProjectRule.java:80)
      	at org.eigenbase.relopt.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:321)
      	at org.eigenbase.relopt.hep.HepPlanner.applyRule(HepPlanner.java:488)
      	at org.eigenbase.relopt.hep.HepPlanner.applyRules(HepPlanner.java:365)
      	at org.eigenbase.relopt.hep.HepPlanner.executeInstruction(HepPlanner.java:258)
      	at org.eigenbase.relopt.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:68)
      	at org.eigenbase.relopt.hep.HepPlanner.executeProgram(HepPlanner.java:179)
      	at org.eigenbase.relopt.hep.HepPlanner.findBestExp(HepPlanner.java:166)
      

        Issue Links

          Activity

          Hide
          ashutoshc Ashutosh Chauhan added a comment - - edited

          Minimal Hive query which results in this is:

          CREATE TABLE myinput1(key int, value int);
          SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b ON ( b.value > 50 ) RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 );
          

          Optiq plan corresponding to this is:

          RelOptUtil.toString(Plan)
            HiveProjectRel(key=[$0])
            HiveJoinRel(condition=[>($6, 50)], joinType=[right])
              HiveJoinRel(condition=[>($6, 50)], joinType=[left])
                HiveTableScanRel(table=[[default.myinput1]])
                HiveTableScanRel(table=[[default.myinput1]])
              HiveTableScanRel(table=[[default.myinput1]])
          
          Show
          ashutoshc Ashutosh Chauhan added a comment - - edited Minimal Hive query which results in this is: CREATE TABLE myinput1(key int , value int ); SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b ON ( b.value > 50 ) RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 ); Optiq plan corresponding to this is: RelOptUtil.toString(Plan) HiveProjectRel(key=[$0]) HiveJoinRel(condition=[>($6, 50)], joinType=[right]) HiveJoinRel(condition=[>($6, 50)], joinType=[left]) HiveTableScanRel(table=[[ default .myinput1]]) HiveTableScanRel(table=[[ default .myinput1]]) HiveTableScanRel(table=[[ default .myinput1]])
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          All of following queries work successfully:

          SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b ON ( b.value > 50 ) RIGHT OUTER JOIN myinput1 c;
          SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 );
          SELECT a.key from myinput1 a JOIN myinput1 b ON ( b.value > 50 ) RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 );
          SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b ON ( b.value > 50 ) JOIN myinput1 c ON ( b.value > 50 );
          
          Show
          ashutoshc Ashutosh Chauhan added a comment - All of following queries work successfully: SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b ON ( b.value > 50 ) RIGHT OUTER JOIN myinput1 c; SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 ); SELECT a.key from myinput1 a JOIN myinput1 b ON ( b.value > 50 ) RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 ); SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b ON ( b.value > 50 ) JOIN myinput1 c ON ( b.value > 50 );
          Hide
          julianhyde Julian Hyde added a comment -

          I don't agree that the behavior of RelOptUtil.eq is wrong. When you have a RexInputRef referencing a field, its type must be IDENTICAL to that field, and that is what eq checks.

          However, something is wrong, probably in one of the rules. I will try to reproduce in pure Calcite.

          Show
          julianhyde Julian Hyde added a comment - I don't agree that the behavior of RelOptUtil.eq is wrong. When you have a RexInputRef referencing a field, its type must be IDENTICAL to that field, and that is what eq checks. However, something is wrong, probably in one of the rules. I will try to reproduce in pure Calcite.
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Julian Hyde Feel free to update the description as you see fit. I might have jumped to solution without understanding the problem here.
          But given that similar other queries pass and this particular one fails gives a strong suspicion that there is something wrong either in Calcite framework or one of its rule.

          Show
          ashutoshc Ashutosh Chauhan added a comment - Julian Hyde Feel free to update the description as you see fit. I might have jumped to solution without understanding the problem here. But given that similar other queries pass and this particular one fails gives a strong suspicion that there is something wrong either in Calcite framework or one of its rule.
          Hide
          julianhyde Julian Hyde added a comment -

          Can you check whether Hive's AST-to-RelNode translator is doing the right thing?

          Note that when translating

          SELECT a.key from myinput1 a
          LEFT OUTER JOIN myinput1 b ON ( b.value > 50 )
          RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 );
          

          The "b.value" in the first ON clause is a different expression than the "b.value" in the second ON clause. The first might be not-nullable (if value is a not-nullable column), while the second is nullable (because it came from the RHS of a left join).

          I am still trying to reproduce this in Calcite (and running into other issues, namely rules generating an infinite network of project on project on project...), but please check that Hive's translator is doing the right thing.

          Show
          julianhyde Julian Hyde added a comment - Can you check whether Hive's AST-to-RelNode translator is doing the right thing? Note that when translating SELECT a.key from myinput1 a LEFT OUTER JOIN myinput1 b ON ( b.value > 50 ) RIGHT OUTER JOIN myinput1 c ON ( b.value > 50 ); The "b.value" in the first ON clause is a different expression than the "b.value" in the second ON clause. The first might be not-nullable (if value is a not-nullable column), while the second is nullable (because it came from the RHS of a left join). I am still trying to reproduce this in Calcite (and running into other issues, namely rules generating an infinite network of project on project on project...), but please check that Hive's translator is doing the right thing.
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          We dont have any logic w.r.t nullability of types while generating calcite plan. In Hive, all columns are always nullable.
          cc: Laljo John Pullokkaran If he knows more.

          Show
          ashutoshc Ashutosh Chauhan added a comment - We dont have any logic w.r.t nullability of types while generating calcite plan. In Hive, all columns are always nullable. cc: Laljo John Pullokkaran If he knows more.
          Hide
          julianhyde Julian Hyde added a comment -

          I finally have a repro case for this. It involves a variant of PushFilterIntoJoinRule that will not allow predicates into inner joins, only into left, right and full outer joins. This is similar to what Hive does, and explains why this occurred in Hive but not pure Calcite.

          Filters are initially pushed down into the ON clause, then kicked back up. At some point we forget to fix up the RexInputRefs in the condition to match whether that field is nullable.

          Show
          julianhyde Julian Hyde added a comment - I finally have a repro case for this. It involves a variant of PushFilterIntoJoinRule that will not allow predicates into inner joins, only into left, right and full outer joins. This is similar to what Hive does, and explains why this occurred in Hive but not pure Calcite. Filters are initially pushed down into the ON clause, then kicked back up. At some point we forget to fix up the RexInputRefs in the condition to match whether that field is nullable.
          Hide
          julianhyde Julian Hyde added a comment -
          Show
          julianhyde Julian Hyde added a comment - I have posted the repro case at https://github.com/julianhyde/incubator-calcite/tree/calcite-448 .
          Hide
          julianhyde Julian Hyde added a comment -
          Show
          julianhyde Julian Hyde added a comment - A patch is now available in https://github.com/julianhyde/incubator-calcite/tree/calcite-448 .
          Show
          julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/011a6b4c .
          Hide
          julianhyde Julian Hyde added a comment -

          Closing now that 1.0.0-incubating has been released.

          Show
          julianhyde Julian Hyde added a comment - Closing now that 1.0.0-incubating has been released.

            People

            • Assignee:
              julianhyde Julian Hyde
              Reporter:
              ashutoshc Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development