Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-507

Add Vagrant files to enable testing of data storages: mysql, postgresql, mongo, etc

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0-incubating
    • Fix Version/s: 1.3.0-incubating
    • Component/s: None
    • Labels:

      Description

      As discussed in email list, it makes sense to implement Vagrant (or similar) VM machines to simplify testing of the downstream executors.

      Currently it is not clear how to configure mysql/mongo/etc.

      Not sure if it should be a set of separate configurations or a single configuration with all the batteries included.

      For instance, Vagrant allows to configure a set of machines, while still providing a way to start just a subset.

        Issue Links

          Activity

          Hide
          vladimirsitnikov Vladimir Sitnikov added a comment - - edited

          Preparation:
          1) Clone https://github.com/vlsi/calcite-test-dataset.git at the same level as calcite repository.
          For instance:

          code
            +-- calcite
            +-- calcite-test-dataset
          

          Note: integration tests search for ../calcite-test-dataset or ../../calcite-test-dataset. You can specify full path via calcite.test.dataset system property.

          2) Build and start the VM: cd calcite-test-dataset && mvn install

          Suggested work flow:

          • To execute regular unit tests (does not require external data): no change. mvn test or mvn install.
          • To execute all tests, for all the DBs: mvn verify -Pit. it stands for "integration-test". mvn install -Pit works as well.
          • To execute just tests for external DBs, excluding unit tests: mvn -Dtest=foo -DfailIfNoTests=false -Pit verify
          • To execute just MongoDB tests: cd mongo; mvn verify -Pit

          From within IDE:

          • To execute regular unit tests: no change.
          • To execute MongoDB tests: run MongoAdapterIT.java as usual (no additional properties are required)
          • To execute MySQL tests: run JdbcTest and JdbcAdapterTest with setting -Dcalcite.test.db=mysql

          Technically:

          • Tests with external data are executed at maven's integration-test phase
          • We do not currently use pre-integration-test/post-integration-test, however we could use that in future
          • The verification of build pass/failure is performed at verify phase

          TODO:

          • Need to pick the way to run tests against file-based databases (e.g. H2). File path is required in JDBC URL, so we need to fix the path. That implies we need to check out test dataset at predefined location (../test-dataset?).
          • Need to think if we should activate "slow tests" by default in it profile, so the running of full suite would be just mvn install -Pit
          Show
          vladimirsitnikov Vladimir Sitnikov added a comment - - edited Preparation: 1) Clone https://github.com/vlsi/calcite-test-dataset.git at the same level as calcite repository. For instance: code +-- calcite +-- calcite-test-dataset Note: integration tests search for ../calcite-test-dataset or ../../calcite-test-dataset . You can specify full path via calcite.test.dataset system property. 2) Build and start the VM: cd calcite-test-dataset && mvn install Suggested work flow: To execute regular unit tests (does not require external data): no change. mvn test or mvn install . To execute all tests, for all the DBs: mvn verify -Pit . it stands for "integration-test". mvn install -Pit works as well. To execute just tests for external DBs, excluding unit tests: mvn -Dtest=foo -DfailIfNoTests=false -Pit verify To execute just MongoDB tests: cd mongo; mvn verify -Pit From within IDE: To execute regular unit tests: no change. To execute MongoDB tests: run MongoAdapterIT.java as usual (no additional properties are required) To execute MySQL tests: run JdbcTest and JdbcAdapterTest with setting -Dcalcite.test.db=mysql Technically: Tests with external data are executed at maven's integration-test phase We do not currently use pre-integration-test / post-integration-test , however we could use that in future The verification of build pass/failure is performed at verify phase TODO: Need to pick the way to run tests against file-based databases (e.g. H2). File path is required in JDBC URL, so we need to fix the path. That implies we need to check out test dataset at predefined location ( ../test-dataset ?). Need to think if we should activate "slow tests" by default in it profile, so the running of full suite would be just mvn install -Pit
          Hide
          vladimirsitnikov Vladimir Sitnikov added a comment -

          PR is updated:
          Mongo – FAIL (recent changes to traits broke it somehow)

          MySQL – pass
          PostgresSQL – pass
          H2 – pass

          Here's how Mongo fails:

          testSelectWhere(org.apache.calcite.test.MongoAdapterIT)  Time elapsed: 1.322 sec  <<< FAILURE!
          java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#226:LogicalProject.NONE.[](input=rel#218:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)]
          	at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108)
          	at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154)
          	at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694)
          	at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510)
          	at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263)
          	at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116)
          	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
          	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
          	at org.apache.calcite.tools.Programs$5.run(Programs.java:272)
          	at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320)
          	at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142)
          	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:274)
          	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188)
          	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:507)
          	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:403)
          	at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:372)
          	at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174)
          	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471)
          	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434)
          	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117)
          	at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:448)
          	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1145)
          	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1227)
          	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1222)
          	at org.apache.calcite.test.MongoAdapterIT.testSelectWhere(MongoAdapterIT.java:331)
          
          Show
          vladimirsitnikov Vladimir Sitnikov added a comment - PR is updated: Mongo – FAIL (recent changes to traits broke it somehow) MySQL – pass PostgresSQL – pass H2 – pass Here's how Mongo fails: testSelectWhere(org.apache.calcite.test.MongoAdapterIT) Time elapsed: 1.322 sec <<< FAILURE! java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#226:LogicalProject.NONE.[](input=rel#218:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)] at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108) at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154) at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694) at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510) at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263) at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795) at org.apache.calcite.tools.Programs$5.run(Programs.java:272) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:274) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:507) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:403) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:372) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:448) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1145) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1227) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1222) at org.apache.calcite.test.MongoAdapterIT.testSelectWhere(MongoAdapterIT.java:331)
          Hide
          julianhyde Julian Hyde added a comment -

          I've just run regular and integration tests on your mongo_tests_fixup branch and everything works. I rebased the branch to the latest master and everything works also. Nice work.

          Can you please add a section to the HOWTO, then I can check it all in? Or indeed you can check in.

          Show
          julianhyde Julian Hyde added a comment - I've just run regular and integration tests on your mongo_tests_fixup branch and everything works. I rebased the branch to the latest master and everything works also. Nice work. Can you please add a section to the HOWTO, then I can check it all in? Or indeed you can check in.
          Hide
          vladimirsitnikov Vladimir Sitnikov added a comment -

          Julian Hyde, can you please post your `cd mongodb && mvn verify -Pit` log?

          Can you please proof-read the update in HOWTO.md ( https://github.com/apache/incubator-calcite/pull/50/files#diff-a12873c54f48832b960cf2ced5eef49e )?

          I get the following as I rebase to the latest master (see updated https://github.com/apache/incubator-calcite/pull/50):

          Failed tests: 
            MongoAdapterIT.testDate:739 Internal error: Error while applying rule MongoProjectRule, args [rel#258:LogicalProject.NONE.[](input=rel#168:Subset#0.MONGO.[],EXPR$0=CAST(ITEM($0, 'date')):DATE)]
            MongoAdapterIT.testDistinctCount:585 Internal error: Error while applying rule MongoProjectRule, args [rel#721:LogicalProject.NONE.[](input=rel#710:Subset#8.NONE.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
            MongoAdapterIT.testDistinctCountOrderBy:617 Internal error: Error while applying rule MongoProjectRule, args [rel#620:LogicalProject.NONE.[](input=rel#167:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
            MongoAdapterIT.testFilter:653 Internal error: Error while applying rule MongoProjectRule, args [rel#231:LogicalProject.NONE.[](input=rel#225:Subset#2.NONE.[],STATE=$1,CITY=$0)]
            MongoAdapterIT.testFilterRedundant:314 Internal error: Error while applying rule MongoProjectRule, args [rel#314:LogicalProject.NONE.[](input=rel#311:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)]
            MongoAdapterIT.testFilterReversed:670 Internal error: Error while applying rule MongoProjectRule, args [rel#240:LogicalProject.NONE.[](input=rel#234:Subset#2.NONE.[],STATE=$1,CITY=$0)]
            MongoAdapterIT.testFilterSortDesc:264 Internal error: Error while applying rule MongoProjectRule, args [rel#322:LogicalProject.NONE.[](input=rel#296:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)]
            MongoAdapterIT.testGroupByHaving:498 Internal error: Error while applying rule MongoFilterRule, args [rel#316:LogicalFilter.NONE.[](input=rel#313:Subset#2.NONE.[],condition=>($1, 1500))]
            MongoAdapterIT.testGroupByMinMaxSum:550 Internal error: Error while applying rule MongoProjectRule, args [rel#274:LogicalProject.NONE.[](input=rel#268:Subset#3.NONE.[],C=$1,STATE=$0,MIN_POP=$2,MAX_POP=$3,SUM_POP=$4)]
            MongoAdapterIT.testGroupByOneColumn:463 Internal error: Error while applying rule MongoProjectRule, args [rel#250:LogicalProject.NONE.[](input=rel#157:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
            MongoAdapterIT.testGroupByOneColumnNotProjected:445 Internal error: Error while applying rule MongoProjectRule, args [rel#261:LogicalProject.NONE.[](input=rel#254:Subset#2.NONE.[],EXPR$0=$1)]
            MongoAdapterIT.testGroupByOneColumnReversed:481 Internal error: Error while applying rule MongoProjectRule, args [rel#298:LogicalProject.NONE.[](input=rel#290:Subset#2.NONE.[],C=$1,STATE=$0)]
            MongoAdapterIT.testGroupComposite:567 Internal error: Error while applying rule MongoProjectRule, args [rel#251:LogicalProject.NONE.[](input=rel#243:Subset#3.NONE.[],C=$2,STATE=$0,CITY=$1)]
            MongoAdapterIT.testInPlan:359 Internal error: Error while applying rule MongoProjectRule, args [rel#235:LogicalProject.NONE.[](input=rel#228:Subset#2.NONE.[],store_id=$0,store_name=$1)]
            MongoAdapterIT.testLimit:214 Internal error: Error while applying rule MongoProjectRule, args [rel#278:LogicalProject.NONE.[](input=rel#163:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
            MongoAdapterIT.testOffsetLimit:200 Internal error: Error while applying rule MongoProjectRule, args [rel#197:LogicalProject.NONE.[](input=rel#156:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
            MongoAdapterIT.testProject:639 Internal error: Error while applying rule MongoProjectRule, args [rel#179:LogicalProject.NONE.[](input=rel#169:Subset#0.MONGO.[],CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
            MongoAdapterIT.testSelectWhere:331 Internal error: Error while applying rule MongoProjectRule, args [rel#196:LogicalProject.NONE.[](input=rel#192:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)]
            MongoAdapterIT.testSort:170 Internal error: Error while applying rule MongoProjectRule, args [rel#195:LogicalProject.NONE.[](input=rel#190:Subset#1.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)]
            MongoAdapterIT.testSortLimit:183 Internal error: Error while applying rule MongoProjectRule, args [rel#200:LogicalProject.NONE.[](input=rel#172:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
            MongoAdapterIT.testUnionPlan:278 Internal error: Error while applying rule MongoProjectRule, args [rel#270:LogicalProject.NONE.[](input=rel#265:Subset#4.NONE.[],product_id=$0)]
            MongoAdapterIT.testZips:410 Internal error: Error while applying rule MongoProjectRule, args [rel#621:LogicalProject.NONE.[](input=rel#173:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
          
          
          
          Tests run: 28, Failures: 22, Errors: 0, Skipped: 5

          The error is like

          java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#7:LogicalProject.NONE.[](input=rel#6:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
          	at org.apache.calcite.util.Util.newInternal(Util.java:743)
          	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
          	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
          	at org.apache.calcite.tools.Programs$5.run(Programs.java:272)
          	at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320)
          	at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142)
          	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:280)
          	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188)
          	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:521)
          	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:417)
          	at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:386)
          	at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174)
          	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471)
          	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434)
          	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117)
          	at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:450)
          	at org.apache.calcite.test.CalciteAssert$AssertQuery.runs(CalciteAssert.java:1176)
          	at org.apache.calcite.test.MongoAdapterIT.testOffsetLimit(MongoAdapterIT.java:200)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
          	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
          	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
          	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
          
          Caused by: java.lang.AssertionError
          	at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108)
          	at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154)
          	at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694)
          	at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510)
          	at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263)
          	at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116)
          	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
          	... 42 more
          
          Show
          vladimirsitnikov Vladimir Sitnikov added a comment - Julian Hyde , can you please post your `cd mongodb && mvn verify -Pit` log? Can you please proof-read the update in HOWTO.md ( https://github.com/apache/incubator-calcite/pull/50/files#diff-a12873c54f48832b960cf2ced5eef49e )? I get the following as I rebase to the latest master (see updated https://github.com/apache/incubator-calcite/pull/50): Failed tests: MongoAdapterIT.testDate:739 Internal error: Error while applying rule MongoProjectRule, args [rel#258:LogicalProject.NONE.[](input=rel#168:Subset#0.MONGO.[],EXPR$0=CAST(ITEM($0, 'date')):DATE)] MongoAdapterIT.testDistinctCount:585 Internal error: Error while applying rule MongoProjectRule, args [rel#721:LogicalProject.NONE.[](input=rel#710:Subset#8.NONE.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testDistinctCountOrderBy:617 Internal error: Error while applying rule MongoProjectRule, args [rel#620:LogicalProject.NONE.[](input=rel#167:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testFilter:653 Internal error: Error while applying rule MongoProjectRule, args [rel#231:LogicalProject.NONE.[](input=rel#225:Subset#2.NONE.[],STATE=$1,CITY=$0)] MongoAdapterIT.testFilterRedundant:314 Internal error: Error while applying rule MongoProjectRule, args [rel#314:LogicalProject.NONE.[](input=rel#311:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)] MongoAdapterIT.testFilterReversed:670 Internal error: Error while applying rule MongoProjectRule, args [rel#240:LogicalProject.NONE.[](input=rel#234:Subset#2.NONE.[],STATE=$1,CITY=$0)] MongoAdapterIT.testFilterSortDesc:264 Internal error: Error while applying rule MongoProjectRule, args [rel#322:LogicalProject.NONE.[](input=rel#296:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)] MongoAdapterIT.testGroupByHaving:498 Internal error: Error while applying rule MongoFilterRule, args [rel#316:LogicalFilter.NONE.[](input=rel#313:Subset#2.NONE.[],condition=>($1, 1500))] MongoAdapterIT.testGroupByMinMaxSum:550 Internal error: Error while applying rule MongoProjectRule, args [rel#274:LogicalProject.NONE.[](input=rel#268:Subset#3.NONE.[],C=$1,STATE=$0,MIN_POP=$2,MAX_POP=$3,SUM_POP=$4)] MongoAdapterIT.testGroupByOneColumn:463 Internal error: Error while applying rule MongoProjectRule, args [rel#250:LogicalProject.NONE.[](input=rel#157:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testGroupByOneColumnNotProjected:445 Internal error: Error while applying rule MongoProjectRule, args [rel#261:LogicalProject.NONE.[](input=rel#254:Subset#2.NONE.[],EXPR$0=$1)] MongoAdapterIT.testGroupByOneColumnReversed:481 Internal error: Error while applying rule MongoProjectRule, args [rel#298:LogicalProject.NONE.[](input=rel#290:Subset#2.NONE.[],C=$1,STATE=$0)] MongoAdapterIT.testGroupComposite:567 Internal error: Error while applying rule MongoProjectRule, args [rel#251:LogicalProject.NONE.[](input=rel#243:Subset#3.NONE.[],C=$2,STATE=$0,CITY=$1)] MongoAdapterIT.testInPlan:359 Internal error: Error while applying rule MongoProjectRule, args [rel#235:LogicalProject.NONE.[](input=rel#228:Subset#2.NONE.[],store_id=$0,store_name=$1)] MongoAdapterIT.testLimit:214 Internal error: Error while applying rule MongoProjectRule, args [rel#278:LogicalProject.NONE.[](input=rel#163:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testOffsetLimit:200 Internal error: Error while applying rule MongoProjectRule, args [rel#197:LogicalProject.NONE.[](input=rel#156:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testProject:639 Internal error: Error while applying rule MongoProjectRule, args [rel#179:LogicalProject.NONE.[](input=rel#169:Subset#0.MONGO.[],CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testSelectWhere:331 Internal error: Error while applying rule MongoProjectRule, args [rel#196:LogicalProject.NONE.[](input=rel#192:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)] MongoAdapterIT.testSort:170 Internal error: Error while applying rule MongoProjectRule, args [rel#195:LogicalProject.NONE.[](input=rel#190:Subset#1.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)] MongoAdapterIT.testSortLimit:183 Internal error: Error while applying rule MongoProjectRule, args [rel#200:LogicalProject.NONE.[](input=rel#172:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testUnionPlan:278 Internal error: Error while applying rule MongoProjectRule, args [rel#270:LogicalProject.NONE.[](input=rel#265:Subset#4.NONE.[],product_id=$0)] MongoAdapterIT.testZips:410 Internal error: Error while applying rule MongoProjectRule, args [rel#621:LogicalProject.NONE.[](input=rel#173:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] Tests run: 28, Failures: 22, Errors: 0, Skipped: 5 The error is like java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#7:LogicalProject.NONE.[](input=rel#6:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] at org.apache.calcite.util.Util.newInternal(Util.java:743) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795) at org.apache.calcite.tools.Programs$5.run(Programs.java:272) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:280) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:521) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:417) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:386) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:450) at org.apache.calcite.test.CalciteAssert$AssertQuery.runs(CalciteAssert.java:1176) at org.apache.calcite.test.MongoAdapterIT.testOffsetLimit(MongoAdapterIT.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) Caused by: java.lang.AssertionError at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108) at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154) at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694) at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510) at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263) at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) ... 42 more
          Hide
          julianhyde Julian Hyde added a comment -

          I only ran `mvn verify -Pit` from the root directory. It seemed to work, but I no longer have the output.

          As you know I planned to freeze for the release today. I don't think I'll get to it before then.

          Show
          julianhyde Julian Hyde added a comment - I only ran `mvn verify -Pit` from the root directory. It seemed to work, but I no longer have the output. As you know I planned to freeze for the release today. I don't think I'll get to it before then.
          Hide
          vladimirsitnikov Vladimir Sitnikov added a comment -

          Well, root build fails in exactly the similar way:

          Tests run: 28, Failures: 22, Errors: 0, Skipped: 5
          
          [INFO] 
          [INFO] --- maven-source-plugin:2.4:jar-no-fork (attach-sources) @ calcite-mongodb ---
          [INFO] 
          [INFO] --- maven-source-plugin:2.4:test-jar-no-fork (attach-sources) @ calcite-mongodb ---
          [INFO] 
          [INFO] --- maven-failsafe-plugin:2.18:verify (failsafe-verify) @ calcite-mongodb ---
          [INFO] Failsafe report directory: /Users/vladimirsitnikov/Documents/code/incubator-optiq/mongodb/target/failsafe-reports
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Calcite ............................................ SUCCESS [  2.408 s]
          [INFO] Calcite Avatica .................................... SUCCESS [  3.388 s]
          [INFO] Calcite Avatica Server ............................. SUCCESS [  0.756 s]
          [INFO] Calcite Linq4j ..................................... SUCCESS [  2.890 s]
          [INFO] Calcite Core ....................................... SUCCESS [04:21 min]
          [INFO] Calcite Examples ................................... SUCCESS [  0.463 s]
          [INFO] Calcite Example CSV ................................ SUCCESS [  2.994 s]
          [INFO] Calcite MongoDB .................................... FAILURE [  4.082 s]
          [INFO] Calcite Plus ....................................... SKIPPED
          [INFO] Calcite Spark ...................................... SKIPPED
          [INFO] Calcite Splunk ..................................... SKIPPED
          [INFO] Calcite Ubenchmark ................................. SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          
          Show
          vladimirsitnikov Vladimir Sitnikov added a comment - Well, root build fails in exactly the similar way: Tests run: 28, Failures: 22, Errors: 0, Skipped: 5 [INFO] [INFO] --- maven-source-plugin:2.4:jar-no-fork (attach-sources) @ calcite-mongodb --- [INFO] [INFO] --- maven-source-plugin:2.4:test-jar-no-fork (attach-sources) @ calcite-mongodb --- [INFO] [INFO] --- maven-failsafe-plugin:2.18:verify (failsafe-verify) @ calcite-mongodb --- [INFO] Failsafe report directory: /Users/vladimirsitnikov/Documents/code/incubator-optiq/mongodb/target/failsafe-reports [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Calcite ............................................ SUCCESS [ 2.408 s] [INFO] Calcite Avatica .................................... SUCCESS [ 3.388 s] [INFO] Calcite Avatica Server ............................. SUCCESS [ 0.756 s] [INFO] Calcite Linq4j ..................................... SUCCESS [ 2.890 s] [INFO] Calcite Core ....................................... SUCCESS [04:21 min] [INFO] Calcite Examples ................................... SUCCESS [ 0.463 s] [INFO] Calcite Example CSV ................................ SUCCESS [ 2.994 s] [INFO] Calcite MongoDB .................................... FAILURE [ 4.082 s] [INFO] Calcite Plus ....................................... SKIPPED [INFO] Calcite Spark ...................................... SKIPPED [INFO] Calcite Splunk ..................................... SKIPPED [INFO] Calcite Ubenchmark ................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE
          Hide
          julianhyde Julian Hyde added a comment -

          I've made some fixes, and I think this is ready to go in. Vladimir Sitnikov, can you please:

          1. Accept my pull request https://github.com/vlsi/calcite-test-dataset/pull/1 up to and including https://github.com/julianhyde/calcite-test-dataset/commit/78507bf41fd9ff60b6aa120b98112004ed4ed0ae.

          2. Review my branch https://github.com/julianhyde/incubator-calcite/tree/507 (which is a rebased version of https://github.com/vlsi/incubator-calcite/tree/mongo_tests_fixup with fixes).

          3. When master branch is open following the 1.2 release, rebase and commit to master.

          Show
          julianhyde Julian Hyde added a comment - I've made some fixes, and I think this is ready to go in. Vladimir Sitnikov , can you please: 1. Accept my pull request https://github.com/vlsi/calcite-test-dataset/pull/1 up to and including https://github.com/julianhyde/calcite-test-dataset/commit/78507bf41fd9ff60b6aa120b98112004ed4ed0ae . 2. Review my branch https://github.com/julianhyde/incubator-calcite/tree/507 (which is a rebased version of https://github.com/vlsi/incubator-calcite/tree/mongo_tests_fixup with fixes). 3. When master branch is open following the 1.2 release, rebase and commit to master.
          Show
          julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/ca871d37 .
          Hide
          vladimirsitnikov Vladimir Sitnikov added a comment -

          Nice.

          I've just tested current master and it works.

          Show
          vladimirsitnikov Vladimir Sitnikov added a comment - Nice. I've just tested current master and it works.
          Hide
          julianhyde Julian Hyde added a comment -

          I've set up a cron job on my home server to run the full regress twice a day. I can't publish the results, but I'll forward the email if we get a failure.

          Show
          julianhyde Julian Hyde added a comment - I've set up a cron job on my home server to run the full regress twice a day. I can't publish the results, but I'll forward the email if we get a failure.
          Hide
          julianhyde Julian Hyde added a comment -

          Resolved in release 1.3.0-incubating (2015-05-30).

          Show
          julianhyde Julian Hyde added a comment - Resolved in release 1.3.0-incubating (2015-05-30).

            People

            • Assignee:
              julianhyde Julian Hyde
              Reporter:
              vladimirsitnikov Vladimir Sitnikov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development