Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 1.7.0
-
Component/s: None
-
Labels:None
Description
Every second time I call
{mvn -DskipTests clean install}it fails with
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/jhyde/open1/calcite.2/core/src/main/java/org/apache/calcite/sql/parser/SqlParser.java:[24,42] cannot find symbol
symbol: class SqlParserImpl
location: package org.apache.calcite.sql.parser.impl
[ERROR] /home/jhyde/open1/calcite.2/core/src/main/java/org/apache/calcite/sql/parser/SqlParser.java:[201,50] cannot find symbol
symbol: variable SqlParserImpl
location: class org.apache.calcite.sql.parser.SqlParser.ConfigBuilder
[INFO] 2 errors
I think CALCITE-484 broke something; maybe the parser is not generated correctly if at the start of the mvn command (before clean) the parser already exists.
Issue Links
- links to
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I got into the habit of doing mvn clean install after a pull because mvn install doesn't re-generate the parser. What should I use instead?
Can you go with mvn clean && mvn install?
That works. Let's leave this issue open, but mvn clean && mvn install is a good workaround.
By the way, the 2nd run of mvn -o -DskipTests -Dcheckstyle.skip=true package now takes 5.6s on my machine. Nice work!
`mvn clean && mvn install` still failed for me. However, trying again with `mvn clean && mvn install -DskipTests` succeeded.
Which git revision were you running? What OS and JDK? Was your error a compile error involving SqlParserImpl, or a failed test?
SqlParserImpl. OS X 10.10.2. 1.8.0_25-b17
Practically the same as me. Are you running an old maven version? I'm running 3.1.1.
I'm also on maven 3.1.1.
Weird. If it happens again please post the build output.
Turns out a small modification I had locally was the problem. I was able to build this time using only `mvn clean && mvn install`. And I forgot to mention the git revision was 341bdd8 (latest master as of now).
This `mvn clean && mvn package` approach works for me as well. Seems like a simple dependency issue in code generation steps. I'll look into it if I get a few minutes.
Slightly related, it would be good to get pre-, post-commit build automation wired up with Apache Jenkins. Anyone with more experience in Incubator know the appropriate channels to pursue this?
I agree. Can you please log a jira case?
I've been using travis on my branch before committing: https://travis-ci.org/julianhyde/incubator-calcite but it isn't as leak-proof as Apache Jenkins. I know James Taylor has set up some pretty amazing Jenkins integration for Phoenix.
I wish I could take credit, but that's Mujtaba Chohan who did that amazing work.
Mujtaba Chohan would you be able to give me a crash-course in ASF jenkins setup? Did you get it done during incubator, or after graduation?
Nick Dimiduk Sure. Created Phoenix jobs by cloning existing HBase jenkins job. It was done in incubation phase. Here are all the Phoenix jobs https://builds.apache.org/view/H-L/view/Phoenix/. BTW sorry missed reading your comment earlier.
I've logged CALCITE-623 to track this. Nick Dimiduk, Mujtaba Chohan, James Taylor, please follow that issue and let's continue discussion there.
I think this is ultimately boiling down to the profile activation check happening before the clean phase runs. The profile isn't activated because core/target/generated-sources/fmpp/javacc/Parser.jj exists, the clean destroys all of target, fmpp doesn't get re-run, and then the javacc plugin also skips itself because its source directory (the fmpp output) doesn't exist.
I can understand the worth in trying to get incremental compliation and faster builds, but it really bothers me when it comes at the cost of breaking the expected semantics of the build tool. I'll see if there's some kind of trick I can find to work around this, but, if I can't figure something out, I'm going to suggest we remove the activation block (and run it all the time).
Oh, I think this is actually a Maven bug. The documentation definitely says that this should work, and upgrading to 3.3.9 does fix it.
I will find the necessary minimum version to get this working and add a maven min-version enforcer statement for this.
Argh, apparently it was just circumstantial.
Alright, I have what I view as a fix here. We move away from the current broken approach (broken, in the sense of what is expected of Maven projects), and move the profiles that are necessary to make the incremental compilation stuff working active by default with the ability to disable them. As such, this should let those whom this is very important to still use benefit from it.
The following profiles can be disable on the command line to avoid the re-generation generate-parser generate-version-properties. Or, in context, mvn package -P'!generate-parser,!generate-version-properties'
mvn clean verify
[INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:16 min [INFO] Finished at: 2016-02-12T00:10:05-05:00 [INFO] Final Memory: 79M/623M [INFO] ------------------------------------------------------------------------
mvn verify -P'!generate-parser,!generate-version-properties'
[INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 04:24 min [INFO] Finished at: 2016-02-12T00:17:07-05:00 [INFO] Final Memory: 49M/627M [INFO] ------------------------------------------------------------------------
Double commit on this one; realized once I pushed that I needed to update the website too.
https://git1-us-west.apache.org/repos/asf?p=calcite.git;a=commit;h=a0b607ade51d2fcd6937f187da2d3691004c03d5 and https://git1-us-west.apache.org/repos/asf?p=calcite.git;a=commit;h=96fd5b7cf36f7893458fbbfc5d716f9202e8407c
Josh Elser, Now mvn -P it,it-oracle install fails for me.
Julian Hyde, ok, I'll look at this today. Feel free to revert if it inhibits you today.
Julian Hyde, I'm just getting some test failures.
Tests in error:
testJoinPlan(org.apache.calcite.test.JdbcAdapterTest): exception while executing [SELECT T1."brand_name"
FROM "foodmart"."product" AS T1
INNER JOIN "foodmart"."product_class" AS T2
ON T1."product_class_id" = T2."product_class_id"
WHERE T2."product_department" = 'Frozen Foods'
OR T2."product_department" = 'Baking Goods'
AND T1."brand_name" <> 'King']
testUnionPlan(org.apache.calcite.test.JdbcAdapterTest): exception while executing [explain plan for select * from "sales_fact_1997"
union all
select * from "sales_fact_1998"]
testMetadataTables(org.apache.calcite.test.JdbcAdapterTest): org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)
testInPlan(org.apache.calcite.test.JdbcAdapterTest): exception while executing [select "store_id", "store_name" from "store"
where "store_name" in ('Store 1', 'Store 10', 'Store 11', 'Store 15', 'Store 16', 'Store 24', 'Store 3', 'Store 7')]
testFilterUnionPlan(org.apache.calcite.test.JdbcAdapterTest): exception while executing [select * from (
select * from "sales_fact_1997"
union all
select * from "sales_fact_1998")
where "product_id" = 1]
testSubQueryWithSingleValue(org.apache.calcite.test.JdbcAdapterTest): exception while executing [explain plan for SELECT "full_name" FROM "employee" WHERE "employee_id" = (SELECT "employee_id" FROM "salary")]
testOrderByOnSortedTable2(org.apache.calcite.test.JdbcTest): exception while executing [select "time_id", "the_date" from "time_by_day"
where "time_id" < 370
order by "time_id"]
testOrderByFetch(org.apache.calcite.test.JdbcTest): exception while executing [explain plan for select "store_id", "grocery_sqft" from "store"
where "store_id" < 10
order by 1 fetch first 5 rows only]
testSelfJoinDifferentColumns(org.apache.calcite.test.JdbcTest): exception while executing [select e1."full_name"
from "foodmart"."employee" as e1
join "foodmart"."employee" as e2 on e1."first_name" = e2."last_name"
order by e1."last_name" limit 3]
testLimitOnQueryableTable(org.apache.calcite.test.JdbcTest): exception while executing [select * from "days"
limit 2]
testHavingNot2(org.apache.calcite.test.JdbcTest): exception while executing [select 1
from "store"
group by "store"."store_street_address"
having NOT (sum("store"."grocery_sqft") < 20000)]
testExplain(org.apache.calcite.test.JdbcTest): exception while executing [explain plan for values (1, 'ab')]
testExtract(org.apache.calcite.test.JdbcTest): exception while executing [values extract(year from date '2008-2-23')]
testOrderBy(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 3 order by 2]
testWhereNot(org.apache.calcite.test.JdbcTest): exception while executing [select 1
from "store"
where NOT ("store"."grocery_sqft" < 22000)
group by "store"."store_street_address"
]
testOrderOnSortedTable(org.apache.calcite.test.JdbcTest): exception while executing [select "day"
from "days"
order by "day"]
testGetDateAsString(org.apache.calcite.test.JdbcTest): exception while executing [select min("date") mindate from "foodmart"."currency"]
testOrderByOffsetFetch(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 10
order by 1 offset 2 rows fetch next 5 rows only]
testVarcharEquals(org.apache.calcite.test.JdbcTest): exception while executing [select "lname" from "customer" where "lname" = 'Nowmer']
testModel(org.apache.calcite.test.JdbcTest): exception while executing [select count(*) as c from "foodmart"."time_by_day"]
testHook(org.apache.calcite.test.JdbcTest): exception while executing [SELECT 1]
testTrim(org.apache.calcite.test.JdbcTest): exception while executing [select trim("lname") as "lname" from "customer" where "lname" = 'Nowmer']
testNullableTimestamp2(org.apache.calcite.test.JdbcTest): exception while executing [select "hire_date", "end_date", "birth_date" from "foodmart"."employee" where "employee_id" = 1]
testOrderByNullsFirst(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 3 order by 2 nulls first]
testUnicode(org.apache.calcite.test.JdbcTest): exception while executing [values _UTF16'英国']
testCloneSchema(org.apache.calcite.test.JdbcTest): org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)
testHavingNot(org.apache.calcite.test.JdbcTest): exception while executing [select "store"."store_country" as "c0", "store"."store_state" as "c1", "store"."store_city" as "c2", "store"."store_name" as "c3", "store"."store_type" as "c4", "store"."store_manager" as "c5", "store"."store_sqft" as "c6", "store"."grocery_sqft" as "c7", "store"."frozen_sqft" as "c8", "store"."meat_sqft" as "c9", "store"."coffee_bar" as "c10", "store"."store_street_address" as "c11" from "store" as "store" group by "store"."store_country", "store"."store_state", "store"."store_city", "store"."store_name", "store"."store_type", "store"."store_manager", "store"."store_sqft", "store"."grocery_sqft", "store"."frozen_sqft", "store"."meat_sqft", "store"."coffee_bar", "store"."store_street_address" having NOT((((sum("store"."store_sqft") - sum("store"."grocery_sqft")) < 10000)) ) order by "store"."store_country" ASC NULLS LAST, "store"."store_state" ASC NULLS LAST, "store"."store_city" ASC NULLS LAST, "store"."store_name" ASC NULLS LAST]
testGetDate(org.apache.calcite.test.JdbcTest): org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)
testOrderByNullsLast(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 3 order by 2 nulls last]
testAggregateFilter(org.apache.calcite.test.JdbcTest): exception while executing [select "the_month",
count(*) as "c",
count(*) filter (where "day_of_month" > 20) as "c2"
from "time_by_day" as "time_by_day"
where "time_by_day"."the_year" = 1997
group by "time_by_day"."the_month"
order by "time_by_day"."the_month"]
testAlmostBushy(org.apache.calcite.test.JdbcTest): exception while executing [explain plan including all attributes for select *
from "sales_fact_1997" as s
join "customer" as c using ("customer_id")
join "product" as p using ("product_id")
where c."city" = 'San Francisco'
and p."brand_name" = 'Washington']
testDistinctCountComposite(org.apache.calcite.test.JdbcTest): exception while executing [select "time_by_day"."the_year" as "c0",
count(distinct "sales_fact_1997"."product_id",
"sales_fact_1997"."customer_id") as "m0"
from "time_by_day" as "time_by_day",
"sales_fact_1997" as "sales_fact_1997"
where "sales_fact_1997"."time_id" = "time_by_day"."time_id"
and "time_by_day"."the_year" = 1997
group by "time_by_day"."the_year"]
testCloneGroupBy(org.apache.calcite.test.JdbcTest): exception while executing [select "the_year", count(*) as c, min("the_month") as m
from "foodmart2"."time_by_day"
group by "the_year"
order by 1, 2]
testOrderByDescNullsLast(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 3 order by 2 desc nulls last]
testOrderByCase(org.apache.calcite.test.JdbcTest): exception while executing [select "time_by_day"."the_year" as "c0" from "time_by_day" as "time_by_day" group by "time_by_day"."the_year" order by CASE WHEN "time_by_day"."the_year" IS NULL THEN 1 ELSE 0 END, "time_by_day"."the_year" ASC]
testOrderByCast(org.apache.calcite.test.JdbcTest): exception while executing [select "customer_id", "postal_code" from "customer"
where "customer_id" < 5
order by cast(substring("postal_code" from 3) as integer) desc]
testOrderByDesc(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 3 order by 2 desc]
testSelfJoinCount(org.apache.calcite.test.JdbcTest): exception while executing [select count(*) as c from "foodmart"."store" as p1 join "foodmart"."store" as p2 using ("store_id")]
testGetTimestampObject(org.apache.calcite.test.JdbcTest): org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)
testModelWithComment(org.apache.calcite.test.JdbcTest): exception while executing [select count(*) as c from "foodmart"."time_by_day"]
testJdbcTimestamp(org.apache.calcite.test.JdbcTest): exception while executing [select count(*) as c from (
select 1 from "foodmart"."employee" as e1
where "hire_date" < TIMESTAMP '1996-06-05 00:00:00'
and "gender" = 'F')]
testOrderByVarious(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 3 order by 2 ]
testDistinctCount2(org.apache.calcite.test.JdbcTest): exception while executing [explain plan for select cast("unit_sales" as integer) as "u",
count(distinct "sales_fact_1997"."customer_id") as "m0"
from "sales_fact_1997" as "sales_fact_1997"
group by "unit_sales"]
testOrderSorted(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id"
from "store"
order by "store_id" limit 3]
testJoinCorrelatedScalarSubquery(org.apache.calcite.test.JdbcTest): exception while executing [select e.employee_id, d.department_id from employee e, department d where e.department_id = d.department_id and e.salary > (select avg(e2.salary) from employee e2 where e2.store_id = e.store_id)]
testRunSort(org.apache.calcite.test.JdbcTest): Error while executing command net.hydromatic.quidem.Quidem$CompositeCommand@65465610
testDialect(org.apache.calcite.test.JdbcTest): exception while executing [select count(*) as c from "foodmart"."employee" as e1
where "first_name" = 'abcde'
and "gender" = 'F']
testOrderByDescNullsFirst(org.apache.calcite.test.JdbcTest): exception while executing [select "store_id", "grocery_sqft" from "store"
where "store_id" < 3 order by 2 desc nulls first]
testJanino169(org.apache.calcite.test.JdbcTest): exception while executing [select "time_id" from "foodmart"."time_by_day" as "t"
]
testSimple(org.apache.calcite.test.JdbcTest): exception while executing [SELECT 1]
testOrderByOnSortedTable(org.apache.calcite.test.JdbcTest): exception while executing [explain plan for select * from "time_by_day"
order by "time_id"]
testNullableTimestamp(org.apache.calcite.test.JdbcTest): exception while executing [select "hire_date", "end_date", "birth_date" from "foodmart"."employee" where "employee_id" = 1]
testSchemaCaching(org.apache.calcite.test.JdbcTest): org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (IO Error: The Network Adapter could not establish the connection)
testSelfJoin(org.apache.calcite.test.JdbcTest): exception while executing [select count(*) as c from (
select 1 from "foodmart"."employee" as e1
join "foodmart"."employee" as e2 using ("position_title"))]
testDistinctCountSimple(org.apache.calcite.test.JdbcTest): exception while executing [explain plan for select count(distinct "sales_fact_1997"."unit_sales") as "m0"
from "sales_fact_1997" as "sales_fact_1997"]
Can you share the error you ran into?
mvn clean && mvn -DskipTests -Pit install
gives
... [INFO] ------------------------------------------------------------------------ [INFO] Building Calcite Core 1.7.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-checkstyle-plugin:2.12.1:check (validate) @ calcite-core --- [INFO] Starting audit... Audit done. [INFO] [INFO] --- javacc-maven-plugin:2.4:javacc (javacc) @ calcite-core --- [INFO] Skipping non-existing source directory: /home/jhyde/open1/calcite.1/core/target/generated-sources/fmpp [INFO] [INFO] --- hydromatic-resource-maven-plugin:0.5.1:generate-sources (default) @ calcite-core --- [INFO] Creating /home/jhyde/open1/calcite.1/core/target/generated-sources/resgen/org/apache/calcite/runtime/Resources.java [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ calcite-core --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (root-resources) @ calcite-core --- [INFO] Skipping remote-resource generation in this project because it's not the Execution Root [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (non-root-resources) @ calcite-core --- [INFO] [INFO] --- build-helper-maven-plugin:1.9:add-resource (add-resource) @ calcite-core --- [INFO] [INFO] --- build-helper-maven-plugin:1.9:add-test-resource (add-resource) @ calcite-core --- [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ calcite-core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] Copying 4 resources to codegen [INFO] Copying 3 resources [INFO] Copying 3 resources [INFO] Copying 2 resources to META-INF [INFO] Copying 1 resource to META-INF [INFO] [INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ calcite-core --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 974 source files to /home/jhyde/open1/calcite.1/core/target/classes [INFO] /home/jhyde/open1/calcite.1/core/src/main/java/org/apache/calcite/sql/SqlOperator.java: Some input files use or override a deprecated API. [INFO] /home/jhyde/open1/calcite.1/core/src/main/java/org/apache/calcite/sql/SqlOperator.java: Recompile with -Xlint:deprecation for details. [INFO] /home/jhyde/open1/calcite.1/core/src/main/java/org/apache/calcite/sql/SqlLiteral.java: Some input files use unchecked or unsafe operations. [INFO] /home/jhyde/open1/calcite.1/core/src/main/java/org/apache/calcite/sql/SqlLiteral.java: Recompile with -Xlint:unchecked for details. [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/jhyde/open1/calcite.1/core/src/main/java/org/apache/calcite/sql/parser/SqlParser.java:[24,42] cannot find symbol symbol: class SqlParserImpl location: package org.apache.calcite.sql.parser.impl [ERROR] /home/jhyde/open1/calcite.1/core/src/main/java/org/apache/calcite/sql/parser/SqlParser.java:[207,50] cannot find symbol symbol: variable SqlParserImpl location: class org.apache.calcite.sql.parser.SqlParser.ConfigBuilder [INFO] 2 errors
I'm going to back out your changes 96fd5b7 and a0b607a, because if -Pit,it-oracle doesn't work (our integration tests) we are flying blind. You probably get errors when running -Pit,it-oracle because you don't have the necessary oracle/mongo/splunk databases installed.
For what it's worth, the script I run is https://github.com/julianhyde/share/blob/master/tools/calcite-regress.sh. I don't claim that you should run that script, or that you should be able to run that script (I don't have the time to make it run on anyone's hardware but my own), but anyone should be able to run -Pit,it-oracle provided that they have the necessary databases running.
I backed out these changes in the force-push that fixed CALCITE-1046. Let's carry on trying to fix this issue, but let's do it on a branch that can be reviewed first.
Alright, sorry for the pain Julian Hyde.
For what it's worth, the script I run is https://github.com/julianhyde/share/blob/master/tools/calcite-regress.sh. I don't claim that you should run that script, or that you should be able to run that script (I don't have the time to make it run on anyone's hardware but my own), but anyone should be able to run -Pit,it-oracle provided that they have the necessary databases running.
Thanks. I had just pulled down the vagrant VM and thought that was going to run things for me.
My hunch is that this has to do with how activeByDefault works in Maven, but I'll prod things some more and put up a branch instead of directly pushing next time.
My hunch is that this has to do with how activeByDefault works in Maven, but I'll prod things some more and put up a branch instead of directly pushing next time.
Yup. That was it. activeByDefault only activates when no profiles are activated by the user. So, with the it profile, it turned off these others. Simple fix to just make them activated by the lack of a property specified by the user at build time. Will open up a PR so we can vet this more before I merge it in.
Reviewing https://github.com/apache/calcite/pull/196:
- Is the ! part of the property name? Or is there a maven convention I'm not aware of. Seems that it might fall afoul of some shells.
- How about having a property called 'generate' or 'skipGenerate' that controls both generations?
- Can you add an example of its usage to howto.md.
Is the ! part of the property name? Or is there a maven convention I'm not aware of. Seems that it might fall afoul of some shells.
This is a Maven feature. "!" is treated as the absence of that property on the command line. So, for a profile with an activation of <name>!foo</name>, the profile would not run if the user added -Dfoo to their mvn invocation.
How about having a property called 'generate' or 'skipGenerate' that controls both generations?
Easy enough.
Can you add an example of its usage to howto.md.
You read my mind. Will update this.
OK, let me know when you have a PR to review. Vladimir Sitnikov, you might like to review also, since you pushed for efficient builds in the first place.
Julian Hyde, I'm inclined to agree that forcing "mvn clean" and "mvn install" separate might be a bit too unexpected, thus we might want to kill that "optimization", and push freemaker to support incremental compilation better (as in "do not overwrite file if the contents is the same")
Just updated the linked PR.
Vladimir Sitnikov, Long term, I agree. And it might be easier now Freemarker are in the Incubator.
Short term, I agree with Josh Elser's proposal to make the default a little slower, and to add a property that can be used by speed-freaks.
Resolved in release 1.7.0 (2016-03-22).
Any good reason for doing clean install often?
The approach with "maven profiles" does indeed suffer from "mvn clean install not supported" problem.
At the start of mvn clean install the parser exists, so "generate parser profile" is not activated.