Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-2039

AssertionError when pushing project to ProjectableFilterableTable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.14.0
    • Fix Version/s: 1.15.0
    • Component/s: core
    • Labels:
      None

      Description

      When my table implements interface ProjectableFilterableTable, and I execute a query that contains column but also an expression, with assertions enabled (java -ea), I get an AssertionError at Mappings.create(Mappings.java:64).

      The following query (notice the "select 0 as c1, ...")

      select 0 as c1, D101.c1 as c3, D101.c2 as c4, D101.c3 as c5 from (
        select T1000001.Category as c1, T1000001.Revenue as c2,
        T1000001."YEAR" as c3 from (
          select "YEAR", Category, "MONTH", Territory, Quarter, Sub_Category,
          Age_Group, Gender, Country, Revenue, Num_Of_Orders, "DATE" from
            XSchema.HDP) T1000001) D101
      

      causes an exception:

      java.lang.AssertionError
      	at org.apache.calcite.util.mapping.Mappings.create(Mappings.java:64)
      	at org.apache.calcite.rel.core.Project.getMapping(Project.java:277)
      	at org.apache.calcite.rel.core.Project.getMapping(Project.java:257)
      	at org.apache.calcite.rel.rules.ProjectTableScanRule.apply(ProjectTableScanRule.java:100)
      	at org.apache.calcite.rel.rules.ProjectTableScanRule$3.onMatch(ProjectTableScanRule.java:83)
      	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
      	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
      	at org.apache.calcite.tools.Programs$5.run(Programs.java:326)
      	at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:387)
      	at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:188)
      	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:321)
      	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
      	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:786)
      	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:640)
      	at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:610)
      	at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:221)
      	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:603)
      	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
      	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
      	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
              ...
      

      In the Mappings.create(), mappingType is INVERSE_SURJECTION, and "assert sourceCount >= targetCount" throws the exception, because sourceCount is 3, and targetCount is 4.
      At the frame above, in Project.getMapping(), "projects" are: [0, $1, $2, $0] (1st is RexLiteral, others are RexInputRef). At the frame above, in EnumerableProject.getMapping(), getInput().getRowType() has only 3 fields in the table.

      I have a reproduction case (in Java) but it's not within a Calcite standard.

      1. CALCITE-2039.zip
        11 kB
        Alexey Roytman
      2. calcite_2039_eclipse_project.zip
        28 kB
        Alexey Roytman
      3. PlannerExampleTest.java
        8 kB
        Enrico Olivelli
      4. PlannerExampleTest.java
        9 kB
        Enrico Olivelli

        Activity

        Hide
        julianhyde Julian Hyde added a comment -

        Resolved in release 1.15.0 (2017-12-11).

        Show
        julianhyde Julian Hyde added a comment - Resolved in release 1.15.0 (2017-12-11).
        Hide
        julianhyde Julian Hyde added a comment -

        Sorry for gazumping you. When I saw a good test case I couldn't resist jumping in.

        Show
        julianhyde Julian Hyde added a comment - Sorry for gazumping you. When I saw a good test case I couldn't resist jumping in.
        Hide
        michaelmior Michael Mior added a comment -

        Thanks for sorting this one out Julian Hyde! I was trying to get to the bottom of this but you obviously had much more success

        Show
        michaelmior Michael Mior added a comment - Thanks for sorting this one out Julian Hyde ! I was trying to get to the bottom of this but you obviously had much more success
        Hide
        eolivelli Enrico Olivelli added a comment -

        Thank you so much.
        Will give it a try soon.

        Show
        eolivelli Enrico Olivelli added a comment - Thank you so much. Will give it a try soon.
        Hide
        julianhyde Julian Hyde added a comment -

        Fixed in 051809b4.

        Show
        julianhyde Julian Hyde added a comment - Fixed in 051809b4 .
        Hide
        julianhyde Julian Hyde added a comment -

        Thanks for your test case, Enrico Olivelli. Yours uses UPDATE, but I was able to convert it into a test case based on SELECT, ScannableTableTest.testCannotPushProject. Will commit shortly.

        Show
        julianhyde Julian Hyde added a comment - Thanks for your test case, Enrico Olivelli . Yours uses UPDATE, but I was able to convert it into a test case based on SELECT, ScannableTableTest.testCannotPushProject . Will commit shortly.
        Hide
        eolivelli Enrico Olivelli added a comment -

        attaching very simple reproducer

        Show
        eolivelli Enrico Olivelli added a comment - attaching very simple reproducer
        Hide
        eolivelli Enrico Olivelli added a comment -

        this is the plan

        Query: UPDATE MYTABLE set id=7 where id=1

        – Logical Plan
        LogicalTableModify(table=[[x, MYTABLE]], operation=[UPDATE], updateColumnList=[[id]], sourceExpressionList=[[7]], flattened=[true])
        LogicalProject(id=[$0], name=[$1], EXPR$0=[7])
        LogicalFilter(condition=[=($0, 1)])
        EnumerableTableScan(table=[[x, MYTABLE]])

        – Mid Plan
        LogicalTableModify(subset=rel#15:Subset#3.ENUMERABLE.any, table=[[x, MYTABLE]], operation=[UPDATE], updateColumnList=[[id]], sourceExpressionList=[[7]], flattened=[true])
        LogicalProject(subset=rel#12:Subset#2.NONE.any, id=[$0], name=[$1], EXPR$0=[7])
        LogicalFilter(subset=rel#10:Subset#1.NONE.any, condition=[=($0, 1)])
        EnumerableTableScan(subset=rel#8:Subset#0.ENUMERABLE.any, table=[[x, MYTABLE]])

        Show
        eolivelli Enrico Olivelli added a comment - this is the plan Query: UPDATE MYTABLE set id=7 where id=1 – Logical Plan LogicalTableModify(table=[ [x, MYTABLE] ], operation= [UPDATE] , updateColumnList=[ [id] ], sourceExpressionList=[ [7] ], flattened= [true] ) LogicalProject(id= [$0] , name= [$1] , EXPR$0= [7] ) LogicalFilter(condition= [=($0, 1)] ) EnumerableTableScan(table=[ [x, MYTABLE] ]) – Mid Plan LogicalTableModify(subset= rel#15:Subset#3.ENUMERABLE.any , table=[ [x, MYTABLE] ], operation= [UPDATE] , updateColumnList=[ [id] ], sourceExpressionList=[ [7] ], flattened= [true] ) LogicalProject(subset= rel#12:Subset#2.NONE.any , id= [$0] , name= [$1] , EXPR$0= [7] ) LogicalFilter(subset= rel#10:Subset#1.NONE.any , condition= [=($0, 1)] ) EnumerableTableScan(subset= rel#8:Subset#0.ENUMERABLE.any , table=[ [x, MYTABLE] ])
        Hide
        eolivelli Enrico Olivelli added a comment -

        is it possible to include in 1.15 ?
        if there is no workaruond it is a real showstopper for me
        thanks

        Show
        eolivelli Enrico Olivelli added a comment - is it possible to include in 1.15 ? if there is no workaruond it is a real showstopper for me thanks
        Hide
        eolivelli Enrico Olivelli added a comment -

        run using latest master

        Show
        eolivelli Enrico Olivelli added a comment - run using latest master
        Hide
        eolivelli Enrico Olivelli added a comment -

        I am getting the same error, I have a simpler reproducer, I am attaching the .java file, it is self contained

        Show
        eolivelli Enrico Olivelli added a comment - I am getting the same error, I have a simpler reproducer, I am attaching the .java file, it is self contained
        Hide
        alexeyroytman Alexey Roytman added a comment -

        Michael, that's right.

        Show
        alexeyroytman Alexey Roytman added a comment - Michael, that's right.
        Hide
        michaelmior Michael Mior added a comment -

        Thanks for sharing. It's a bit hard to follow what's going in your code, but I'm guessing you want to check over your projection logic to make sure the rows you return have the correct columns projected.

        Show
        michaelmior Michael Mior added a comment - Thanks for sharing. It's a bit hard to follow what's going in your code, but I'm guessing you want to check over your projection logic to make sure the rows you return have the correct columns projected.
        Hide
        alexeyroytman Alexey Roytman added a comment -

        Created, attached.

        Show
        alexeyroytman Alexey Roytman added a comment - Created, attached.
        Hide
        alexeyroytman Alexey Roytman added a comment -

        Michael, I'll create an Eclipse project.

        Show
        alexeyroytman Alexey Roytman added a comment - Michael, I'll create an Eclipse project.
        Hide
        michaelmior Michael Mior added a comment -

        Alexey Roytman If you are able to spend a bit of time creating a self-contained project I can compile and run, that would be helpful. Otherwise, I'll try to get to this myself when I can.

        Show
        michaelmior Michael Mior added a comment - Alexey Roytman If you are able to spend a bit of time creating a self-contained project I can compile and run, that would be helpful. Otherwise, I'll try to get to this myself when I can.
        Hide
        alexeyroytman Alexey Roytman added a comment -

        Michael, it's a part of my maven project with dependencies on calcite 1.14.0, junit 4.11 (and some irrelevant to the test).
        In eclipse I run the junit file I mentioned above... Michael, this is a part of my "not within a Calcite standard" excuse

        Show
        alexeyroytman Alexey Roytman added a comment - Michael, it's a part of my maven project with dependencies on calcite 1.14.0, junit 4.11 (and some irrelevant to the test). In eclipse I run the junit file I mentioned above... Michael, this is a part of my "not within a Calcite standard" excuse
        Hide
        michaelmior Michael Mior added a comment -

        Julian Hyde Sure, assigned to myself. Alexey Roytman Can you explain how you're building the code in the attached zip?

        Show
        michaelmior Michael Mior added a comment - Julian Hyde Sure, assigned to myself. Alexey Roytman Can you explain how you're building the code in the attached zip?
        Hide
        julianhyde Julian Hyde added a comment -

        Michael Mior, do you mind if I assign this one to you?

        Show
        julianhyde Julian Hyde added a comment - Michael Mior , do you mind if I assign this one to you?
        Hide
        alexeyroytman Alexey Roytman added a comment -

        Please tell me if anything is missing from CALCITE-2039.zip. The JUnit is in tests/tests/sql/xcsv/SQLTestCalciteIFxcsv.java file there.

        Show
        alexeyroytman Alexey Roytman added a comment - Please tell me if anything is missing from CALCITE-2039 .zip. The JUnit is in tests/tests/sql/xcsv/SQLTestCalciteIFxcsv.java file there.
        Hide
        alexeyroytman Alexey Roytman added a comment -

        I mean I have some code that reproduces the case, but it definitely looks unlike all the code in the Calcite's tests.
        I'll attach relevant files. But I beg pardon in advance for the style and the way.

        Show
        alexeyroytman Alexey Roytman added a comment - I mean I have some code that reproduces the case, but it definitely looks unlike all the code in the Calcite's tests. I'll attach relevant files. But I beg pardon in advance for the style and the way.
        Hide
        michaelmior Michael Mior added a comment -

        Could you explain what you mean by "not within a Calcite standard"?

        Show
        michaelmior Michael Mior added a comment - Could you explain what you mean by "not within a Calcite standard"?

          People

          • Assignee:
            michaelmior Michael Mior
            Reporter:
            alexeyroytman Alexey Roytman
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development