[CALCITE-507] Add Vagrant files to enable testing of data storages: mysql, postgresql, mongo, etc - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 1.0.0-incubating
Fix Version/s: 1.3.0-incubating
Component/s: None
Labels:
- newbie

Description

As discussed in email list, it makes sense to implement Vagrant (or similar) VM machines to simplify testing of the downstream executors.

Currently it is not clear how to configure mysql/mongo/etc.

Not sure if it should be a set of separate configurations or a single configuration with all the batteries included.

For instance, Vagrant allows to configure a set of machines, while still providing a way to start just a subset.

Issue Links

is depended upon by

CALCITE-579 Regularly and automatically run full test suite on Vagrant VM

Closed

Options
- Show All
- Show Open

Sub-Tasks

Update MongoDB test suite to calcite-test-dataset

Closed

Julian Hyde

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Vladimir Sitnikov added a comment - 09/Feb/15 07:13 - edited

Preparation:
1) Clone https://github.com/vlsi/calcite-test-dataset.git at the same level as calcite repository.
For instance:

code
  +-- calcite
  +-- calcite-test-dataset

Note: integration tests search for ../calcite-test-dataset or ../../calcite-test-dataset. You can specify full path via calcite.test.dataset system property.

2) Build and start the VM: cd calcite-test-dataset && mvn install

Suggested work flow:

To execute regular unit tests (does not require external data): no change. mvn test or mvn install.
To execute all tests, for all the DBs: mvn verify -Pit. it stands for "integration-test". mvn install -Pit works as well.
To execute just tests for external DBs, excluding unit tests: mvn -Dtest=foo -DfailIfNoTests=false -Pit verify
To execute just MongoDB tests: cd mongo; mvn verify -Pit

From within IDE:

To execute regular unit tests: no change.
To execute MongoDB tests: run MongoAdapterIT.java as usual (no additional properties are required)
To execute MySQL tests: run JdbcTest and JdbcAdapterTest with setting -Dcalcite.test.db=mysql

Technically:

Tests with external data are executed at maven's integration-test phase
We do not currently use pre-integration-test/post-integration-test, however we could use that in future
The verification of build pass/failure is performed at verify phase

TODO:

Need to pick the way to run tests against file-based databases (e.g. H2). File path is required in JDBC URL, so we need to fix the path. That implies we need to check out test dataset at predefined location (../test-dataset?).
Need to think if we should activate "slow tests" by default in it profile, so the running of full suite would be just mvn install -Pit

Show

Vladimir Sitnikov added a comment - 09/Feb/15 07:13 - edited Preparation: 1) Clone https://github.com/vlsi/calcite-test-dataset.git at the same level as calcite repository. For instance: code +-- calcite +-- calcite-test-dataset Note: integration tests search for ../calcite-test-dataset or ../../calcite-test-dataset . You can specify full path via calcite.test.dataset system property. 2) Build and start the VM: cd calcite-test-dataset && mvn install Suggested work flow: To execute regular unit tests (does not require external data): no change. mvn test or mvn install . To execute all tests, for all the DBs: mvn verify -Pit . it stands for "integration-test". mvn install -Pit works as well. To execute just tests for external DBs, excluding unit tests: mvn -Dtest=foo -DfailIfNoTests=false -Pit verify To execute just MongoDB tests: cd mongo; mvn verify -Pit From within IDE: To execute regular unit tests: no change. To execute MongoDB tests: run MongoAdapterIT.java as usual (no additional properties are required) To execute MySQL tests: run JdbcTest and JdbcAdapterTest with setting -Dcalcite.test.db=mysql Technically: Tests with external data are executed at maven's integration-test phase We do not currently use pre-integration-test / post-integration-test , however we could use that in future The verification of build pass/failure is performed at verify phase TODO: Need to pick the way to run tests against file-based databases (e.g. H2). File path is required in JDBC URL, so we need to fix the path. That implies we need to check out test dataset at predefined location ( ../test-dataset ?). Need to think if we should activate "slow tests" by default in it profile, so the running of full suite would be just mvn install -Pit

Hide

Permalink

Vladimir Sitnikov added a comment - 10/Feb/15 10:02

PR is updated:
Mongo – FAIL (recent changes to traits broke it somehow)

MySQL – pass
PostgresSQL – pass
H2 – pass

Here's how Mongo fails:

testSelectWhere(org.apache.calcite.test.MongoAdapterIT)  Time elapsed: 1.322 sec  <<< FAILURE!
java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#226:LogicalProject.NONE.[](input=rel#218:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)]
	at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108)
	at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694)
	at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510)
	at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263)
	at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116)
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
	at org.apache.calcite.tools.Programs$5.run(Programs.java:272)
	at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320)
	at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142)
	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:274)
	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188)
	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:507)
	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:403)
	at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:372)
	at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174)
	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471)
	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434)
	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117)
	at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:448)
	at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1145)
	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1227)
	at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1222)
	at org.apache.calcite.test.MongoAdapterIT.testSelectWhere(MongoAdapterIT.java:331)

Show

Vladimir Sitnikov added a comment - 10/Feb/15 10:02 PR is updated: Mongo – FAIL (recent changes to traits broke it somehow) MySQL – pass PostgresSQL – pass H2 – pass Here's how Mongo fails: testSelectWhere(org.apache.calcite.test.MongoAdapterIT) Time elapsed: 1.322 sec <<< FAILURE! java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#226:LogicalProject.NONE.[](input=rel#218:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)] at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108) at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154) at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694) at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510) at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263) at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795) at org.apache.calcite.tools.Programs$5.run(Programs.java:272) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:274) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:507) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:403) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:372) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:448) at org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1145) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainMatches(CalciteAssert.java:1227) at org.apache.calcite.test.CalciteAssert$AssertQuery.explainContains(CalciteAssert.java:1222) at org.apache.calcite.test.MongoAdapterIT.testSelectWhere(MongoAdapterIT.java:331)

Hide

Permalink

Julian Hyde added a comment - 10/Mar/15 08:34

I've just run regular and integration tests on your mongo_tests_fixup branch and everything works. I rebased the branch to the latest master and everything works also. Nice work.

Can you please add a section to the HOWTO, then I can check it all in? Or indeed you can check in.

Show

Julian Hyde added a comment - 10/Mar/15 08:34 I've just run regular and integration tests on your mongo_tests_fixup branch and everything works. I rebased the branch to the latest master and everything works also. Nice work. Can you please add a section to the HOWTO, then I can check it all in? Or indeed you can check in.

Hide

Permalink

Vladimir Sitnikov added a comment - 10/Mar/15 20:52

Julian Hyde, can you please post your `cd mongodb && mvn verify -Pit` log?

Can you please proof-read the update in HOWTO.md ( https://github.com/apache/incubator-calcite/pull/50/files#diff-a12873c54f48832b960cf2ced5eef49e )?

I get the following as I rebase to the latest master (see updated https://github.com/apache/incubator-calcite/pull/50):

Failed tests: 
  MongoAdapterIT.testDate:739 Internal error: Error while applying rule MongoProjectRule, args [rel#258:LogicalProject.NONE.[](input=rel#168:Subset#0.MONGO.[],EXPR$0=CAST(ITEM($0, 'date')):DATE)]
  MongoAdapterIT.testDistinctCount:585 Internal error: Error while applying rule MongoProjectRule, args [rel#721:LogicalProject.NONE.[](input=rel#710:Subset#8.NONE.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
  MongoAdapterIT.testDistinctCountOrderBy:617 Internal error: Error while applying rule MongoProjectRule, args [rel#620:LogicalProject.NONE.[](input=rel#167:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
  MongoAdapterIT.testFilter:653 Internal error: Error while applying rule MongoProjectRule, args [rel#231:LogicalProject.NONE.[](input=rel#225:Subset#2.NONE.[],STATE=$1,CITY=$0)]
  MongoAdapterIT.testFilterRedundant:314 Internal error: Error while applying rule MongoProjectRule, args [rel#314:LogicalProject.NONE.[](input=rel#311:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)]
  MongoAdapterIT.testFilterReversed:670 Internal error: Error while applying rule MongoProjectRule, args [rel#240:LogicalProject.NONE.[](input=rel#234:Subset#2.NONE.[],STATE=$1,CITY=$0)]
  MongoAdapterIT.testFilterSortDesc:264 Internal error: Error while applying rule MongoProjectRule, args [rel#322:LogicalProject.NONE.[](input=rel#296:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)]
  MongoAdapterIT.testGroupByHaving:498 Internal error: Error while applying rule MongoFilterRule, args [rel#316:LogicalFilter.NONE.[](input=rel#313:Subset#2.NONE.[],condition=>($1, 1500))]
  MongoAdapterIT.testGroupByMinMaxSum:550 Internal error: Error while applying rule MongoProjectRule, args [rel#274:LogicalProject.NONE.[](input=rel#268:Subset#3.NONE.[],C=$1,STATE=$0,MIN_POP=$2,MAX_POP=$3,SUM_POP=$4)]
  MongoAdapterIT.testGroupByOneColumn:463 Internal error: Error while applying rule MongoProjectRule, args [rel#250:LogicalProject.NONE.[](input=rel#157:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
  MongoAdapterIT.testGroupByOneColumnNotProjected:445 Internal error: Error while applying rule MongoProjectRule, args [rel#261:LogicalProject.NONE.[](input=rel#254:Subset#2.NONE.[],EXPR$0=$1)]
  MongoAdapterIT.testGroupByOneColumnReversed:481 Internal error: Error while applying rule MongoProjectRule, args [rel#298:LogicalProject.NONE.[](input=rel#290:Subset#2.NONE.[],C=$1,STATE=$0)]
  MongoAdapterIT.testGroupComposite:567 Internal error: Error while applying rule MongoProjectRule, args [rel#251:LogicalProject.NONE.[](input=rel#243:Subset#3.NONE.[],C=$2,STATE=$0,CITY=$1)]
  MongoAdapterIT.testInPlan:359 Internal error: Error while applying rule MongoProjectRule, args [rel#235:LogicalProject.NONE.[](input=rel#228:Subset#2.NONE.[],store_id=$0,store_name=$1)]
  MongoAdapterIT.testLimit:214 Internal error: Error while applying rule MongoProjectRule, args [rel#278:LogicalProject.NONE.[](input=rel#163:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
  MongoAdapterIT.testOffsetLimit:200 Internal error: Error while applying rule MongoProjectRule, args [rel#197:LogicalProject.NONE.[](input=rel#156:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
  MongoAdapterIT.testProject:639 Internal error: Error while applying rule MongoProjectRule, args [rel#179:LogicalProject.NONE.[](input=rel#169:Subset#0.MONGO.[],CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
  MongoAdapterIT.testSelectWhere:331 Internal error: Error while applying rule MongoProjectRule, args [rel#196:LogicalProject.NONE.[](input=rel#192:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)]
  MongoAdapterIT.testSort:170 Internal error: Error while applying rule MongoProjectRule, args [rel#195:LogicalProject.NONE.[](input=rel#190:Subset#1.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)]
  MongoAdapterIT.testSortLimit:183 Internal error: Error while applying rule MongoProjectRule, args [rel#200:LogicalProject.NONE.[](input=rel#172:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
  MongoAdapterIT.testUnionPlan:278 Internal error: Error while applying rule MongoProjectRule, args [rel#270:LogicalProject.NONE.[](input=rel#265:Subset#4.NONE.[],product_id=$0)]
  MongoAdapterIT.testZips:410 Internal error: Error while applying rule MongoProjectRule, args [rel#621:LogicalProject.NONE.[](input=rel#173:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]



Tests run: 28, Failures: 22, Errors: 0, Skipped: 5

The error is like

java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#7:LogicalProject.NONE.[](input=rel#6:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")]
	at org.apache.calcite.util.Util.newInternal(Util.java:743)
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
	at org.apache.calcite.tools.Programs$5.run(Programs.java:272)
	at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320)
	at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142)
	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:280)
	at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188)
	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:521)
	at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:417)
	at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:386)
	at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174)
	at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471)
	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434)
	at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117)
	at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:450)
	at org.apache.calcite.test.CalciteAssert$AssertQuery.runs(CalciteAssert.java:1176)
	at org.apache.calcite.test.MongoAdapterIT.testOffsetLimit(MongoAdapterIT.java:200)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

Caused by: java.lang.AssertionError
	at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108)
	at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154)
	at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694)
	at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510)
	at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263)
	at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116)
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
	... 42 more

Show

Vladimir Sitnikov added a comment - 10/Mar/15 20:52 Julian Hyde , can you please post your `cd mongodb && mvn verify -Pit` log? Can you please proof-read the update in HOWTO.md ( https://github.com/apache/incubator-calcite/pull/50/files#diff-a12873c54f48832b960cf2ced5eef49e )? I get the following as I rebase to the latest master (see updated https://github.com/apache/incubator-calcite/pull/50): Failed tests: MongoAdapterIT.testDate:739 Internal error: Error while applying rule MongoProjectRule, args [rel#258:LogicalProject.NONE.[](input=rel#168:Subset#0.MONGO.[],EXPR$0=CAST(ITEM($0, 'date')):DATE)] MongoAdapterIT.testDistinctCount:585 Internal error: Error while applying rule MongoProjectRule, args [rel#721:LogicalProject.NONE.[](input=rel#710:Subset#8.NONE.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testDistinctCountOrderBy:617 Internal error: Error while applying rule MongoProjectRule, args [rel#620:LogicalProject.NONE.[](input=rel#167:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testFilter:653 Internal error: Error while applying rule MongoProjectRule, args [rel#231:LogicalProject.NONE.[](input=rel#225:Subset#2.NONE.[],STATE=$1,CITY=$0)] MongoAdapterIT.testFilterRedundant:314 Internal error: Error while applying rule MongoProjectRule, args [rel#314:LogicalProject.NONE.[](input=rel#311:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)] MongoAdapterIT.testFilterReversed:670 Internal error: Error while applying rule MongoProjectRule, args [rel#240:LogicalProject.NONE.[](input=rel#234:Subset#2.NONE.[],STATE=$1,CITY=$0)] MongoAdapterIT.testFilterSortDesc:264 Internal error: Error while applying rule MongoProjectRule, args [rel#322:LogicalProject.NONE.[](input=rel#296:Subset#2.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)] MongoAdapterIT.testGroupByHaving:498 Internal error: Error while applying rule MongoFilterRule, args [rel#316:LogicalFilter.NONE.[](input=rel#313:Subset#2.NONE.[],condition=>($1, 1500))] MongoAdapterIT.testGroupByMinMaxSum:550 Internal error: Error while applying rule MongoProjectRule, args [rel#274:LogicalProject.NONE.[](input=rel#268:Subset#3.NONE.[],C=$1,STATE=$0,MIN_POP=$2,MAX_POP=$3,SUM_POP=$4)] MongoAdapterIT.testGroupByOneColumn:463 Internal error: Error while applying rule MongoProjectRule, args [rel#250:LogicalProject.NONE.[](input=rel#157:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testGroupByOneColumnNotProjected:445 Internal error: Error while applying rule MongoProjectRule, args [rel#261:LogicalProject.NONE.[](input=rel#254:Subset#2.NONE.[],EXPR$0=$1)] MongoAdapterIT.testGroupByOneColumnReversed:481 Internal error: Error while applying rule MongoProjectRule, args [rel#298:LogicalProject.NONE.[](input=rel#290:Subset#2.NONE.[],C=$1,STATE=$0)] MongoAdapterIT.testGroupComposite:567 Internal error: Error while applying rule MongoProjectRule, args [rel#251:LogicalProject.NONE.[](input=rel#243:Subset#3.NONE.[],C=$2,STATE=$0,CITY=$1)] MongoAdapterIT.testInPlan:359 Internal error: Error while applying rule MongoProjectRule, args [rel#235:LogicalProject.NONE.[](input=rel#228:Subset#2.NONE.[],store_id=$0,store_name=$1)] MongoAdapterIT.testLimit:214 Internal error: Error while applying rule MongoProjectRule, args [rel#278:LogicalProject.NONE.[](input=rel#163:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testOffsetLimit:200 Internal error: Error while applying rule MongoProjectRule, args [rel#197:LogicalProject.NONE.[](input=rel#156:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testProject:639 Internal error: Error while applying rule MongoProjectRule, args [rel#179:LogicalProject.NONE.[](input=rel#169:Subset#0.MONGO.[],CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testSelectWhere:331 Internal error: Error while applying rule MongoProjectRule, args [rel#196:LogicalProject.NONE.[](input=rel#192:Subset#2.NONE.[],warehouse_id=$0,warehouse_state_province=$1)] MongoAdapterIT.testSort:170 Internal error: Error while applying rule MongoProjectRule, args [rel#195:LogicalProject.NONE.[](input=rel#190:Subset#1.NONE.[],CITY=$0,LONGITUDE=$1,LATITUDE=$2,POP=$3,STATE=$4,ID=$5)] MongoAdapterIT.testSortLimit:183 Internal error: Error while applying rule MongoProjectRule, args [rel#200:LogicalProject.NONE.[](input=rel#172:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] MongoAdapterIT.testUnionPlan:278 Internal error: Error while applying rule MongoProjectRule, args [rel#270:LogicalProject.NONE.[](input=rel#265:Subset#4.NONE.[],product_id=$0)] MongoAdapterIT.testZips:410 Internal error: Error while applying rule MongoProjectRule, args [rel#621:LogicalProject.NONE.[](input=rel#173:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",CITY=CAST(ITEM($0, 'city')):VARCHAR(20) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] Tests run: 28, Failures: 22, Errors: 0, Skipped: 5 The error is like java.lang.AssertionError: Internal error: Error while applying rule MongoProjectRule, args [rel#7:LogicalProject.NONE.[](input=rel#6:Subset#0.MONGO.[],STATE=CAST(ITEM($0, 'state')):VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary",ID=CAST(ITEM($0, '_id')):VARCHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary")] at org.apache.calcite.util.Util.newInternal(Util.java:743) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251) at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795) at org.apache.calcite.tools.Programs$5.run(Programs.java:272) at org.apache.calcite.tools.Programs$SequenceProgram.run(Programs.java:320) at org.apache.calcite.prepare.Prepare.optimize(Prepare.java:142) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:280) at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:188) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:521) at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:417) at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:386) at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:174) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:471) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:434) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:117) at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:450) at org.apache.calcite.test.CalciteAssert$AssertQuery.runs(CalciteAssert.java:1176) at org.apache.calcite.test.MongoAdapterIT.testOffsetLimit(MongoAdapterIT.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) Caused by: java.lang.AssertionError at org.apache.calcite.plan.volcano.RelSubset.<init>(RelSubset.java:108) at org.apache.calcite.plan.volcano.RelSet.getOrCreateSubset(RelSet.java:154) at org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:694) at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:510) at org.apache.calcite.adapter.mongodb.MongoRules$MongoProjectRule.convert(MongoRules.java:263) at org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:116) at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) ... 42 more

Hide

Permalink

Julian Hyde added a comment - 10/Mar/15 21:19

I only ran `mvn verify -Pit` from the root directory. It seemed to work, but I no longer have the output.

As you know I planned to freeze for the release today. I don't think I'll get to it before then.

Show

Julian Hyde added a comment - 10/Mar/15 21:19 I only ran `mvn verify -Pit` from the root directory. It seemed to work, but I no longer have the output. As you know I planned to freeze for the release today. I don't think I'll get to it before then.

Hide

Permalink

Vladimir Sitnikov added a comment - 10/Mar/15 21:59

Well, root build fails in exactly the similar way:

Tests run: 28, Failures: 22, Errors: 0, Skipped: 5

[INFO] 
[INFO] --- maven-source-plugin:2.4:jar-no-fork (attach-sources) @ calcite-mongodb ---
[INFO] 
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (attach-sources) @ calcite-mongodb ---
[INFO] 
[INFO] --- maven-failsafe-plugin:2.18:verify (failsafe-verify) @ calcite-mongodb ---
[INFO] Failsafe report directory: /Users/vladimirsitnikov/Documents/code/incubator-optiq/mongodb/target/failsafe-reports
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Calcite ............................................ SUCCESS [  2.408 s]
[INFO] Calcite Avatica .................................... SUCCESS [  3.388 s]
[INFO] Calcite Avatica Server ............................. SUCCESS [  0.756 s]
[INFO] Calcite Linq4j ..................................... SUCCESS [  2.890 s]
[INFO] Calcite Core ....................................... SUCCESS [04:21 min]
[INFO] Calcite Examples ................................... SUCCESS [  0.463 s]
[INFO] Calcite Example CSV ................................ SUCCESS [  2.994 s]
[INFO] Calcite MongoDB .................................... FAILURE [  4.082 s]
[INFO] Calcite Plus ....................................... SKIPPED
[INFO] Calcite Spark ...................................... SKIPPED
[INFO] Calcite Splunk ..................................... SKIPPED
[INFO] Calcite Ubenchmark ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Show

Vladimir Sitnikov added a comment - 10/Mar/15 21:59 Well, root build fails in exactly the similar way: Tests run: 28, Failures: 22, Errors: 0, Skipped: 5 [INFO] [INFO] --- maven-source-plugin:2.4:jar-no-fork (attach-sources) @ calcite-mongodb --- [INFO] [INFO] --- maven-source-plugin:2.4:test-jar-no-fork (attach-sources) @ calcite-mongodb --- [INFO] [INFO] --- maven-failsafe-plugin:2.18:verify (failsafe-verify) @ calcite-mongodb --- [INFO] Failsafe report directory: /Users/vladimirsitnikov/Documents/code/incubator-optiq/mongodb/target/failsafe-reports [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Calcite ............................................ SUCCESS [ 2.408 s] [INFO] Calcite Avatica .................................... SUCCESS [ 3.388 s] [INFO] Calcite Avatica Server ............................. SUCCESS [ 0.756 s] [INFO] Calcite Linq4j ..................................... SUCCESS [ 2.890 s] [INFO] Calcite Core ....................................... SUCCESS [04:21 min] [INFO] Calcite Examples ................................... SUCCESS [ 0.463 s] [INFO] Calcite Example CSV ................................ SUCCESS [ 2.994 s] [INFO] Calcite MongoDB .................................... FAILURE [ 4.082 s] [INFO] Calcite Plus ....................................... SKIPPED [INFO] Calcite Spark ...................................... SKIPPED [INFO] Calcite Splunk ..................................... SKIPPED [INFO] Calcite Ubenchmark ................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE

Hide

Permalink

Julian Hyde added a comment - 08/Apr/15 23:38

I've made some fixes, and I think this is ready to go in. Vladimir Sitnikov, can you please:

1. Accept my pull request https://github.com/vlsi/calcite-test-dataset/pull/1 up to and including https://github.com/julianhyde/calcite-test-dataset/commit/78507bf41fd9ff60b6aa120b98112004ed4ed0ae.

2. Review my branch https://github.com/julianhyde/incubator-calcite/tree/507 (which is a rebased version of https://github.com/vlsi/incubator-calcite/tree/mongo_tests_fixup with fixes).

3. When master branch is open following the 1.2 release, rebase and commit to master.

Show

Julian Hyde added a comment - 08/Apr/15 23:38 I've made some fixes, and I think this is ready to go in. Vladimir Sitnikov , can you please: 1. Accept my pull request https://github.com/vlsi/calcite-test-dataset/pull/1 up to and including https://github.com/julianhyde/calcite-test-dataset/commit/78507bf41fd9ff60b6aa120b98112004ed4ed0ae . 2. Review my branch https://github.com/julianhyde/incubator-calcite/tree/507 (which is a rebased version of https://github.com/vlsi/incubator-calcite/tree/mongo_tests_fixup with fixes). 3. When master branch is open following the 1.2 release, rebase and commit to master.

Hide

Permalink

Julian Hyde added a comment - 17/Apr/15 19:40

Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/ca871d37.

Show

Julian Hyde added a comment - 17/Apr/15 19:40 Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/ca871d37 .

Hide

Permalink

Vladimir Sitnikov added a comment - 18/Apr/15 09:38

Nice.

I've just tested current master and it works.

Show

Vladimir Sitnikov added a comment - 18/Apr/15 09:38 Nice. I've just tested current master and it works.

Hide

Permalink

Julian Hyde added a comment - 18/Apr/15 15:43

I've set up a cron job on my home server to run the full regress twice a day. I can't publish the results, but I'll forward the email if we get a failure.

Show

Julian Hyde added a comment - 18/Apr/15 15:43 I've set up a cron job on my home server to run the full regress twice a day. I can't publish the results, but I'll forward the email if we get a failure.

Hide

Permalink

Julian Hyde added a comment - 01/Jun/15 18:24

Resolved in release 1.3.0-incubating (2015-05-30).

Show

Julian Hyde added a comment - 01/Jun/15 18:24 Resolved in release 1.3.0-incubating (2015-05-30).

People

Assignee:

Julian Hyde

Reporter:

Vladimir Sitnikov

Votes:

0 Vote for this issue

Watchers:

3 Start watching this issue

Dates

Created:

02/Dec/14 12:09

Updated:

01/Jun/15 18:24

Resolved:

17/Apr/15 19:40

Development

Agile

View on Board