Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0-alpha-1
-
None
Description
When folks "install a JDK" wether they get JDK11 now will depend on the platform (e.g. from homebrew you will get jdk11). For new contributors this results in confusing errors on first run.
Make it so that a non-release build using JDK11 and using our default compiler source/target versions of JDK8 can complete.
Attachments
Attachments
- HBASE-21808.0.patch
- 9 kB
- Sean Busbey
Issue Links
- breaks
-
HBASE-22345 REST Server must have specific version of javax.annotations available at runtime
- Resolved
Activity
on RHEL7.6 yum install java-devel currently still gets 1.8u191. to get jdk11 you have expressly ask via e.g. yum install java-11-openjdk-devel.
example error for the search engines:
[busbey@busbey-centos76 hbase]$ mvn -version Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T11:41:47-07:00) Maven home: /usr/local/share/applications/apache-maven-3.6.0 Java version: 11.0.1, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-11-openjdk-11.0.1.13-3.el7_6.x86_64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-957.el7.x86_64", arch: "amd64", family: "unix" [busbey@busbey-centos76 hbase]$ mvn -DskipTests --batch-mode install [INFO] Scanning for projects... ... [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/busbey/projects/hbase/hbase-protocol-shaded/target/generated-sources/protobuf/java/org/apache/hadoop/hbase/shaded/protobuf/generated/SnapshotProtos.java:[6,18] package javax.annotation does not exist [ERROR] /home/busbey/projects/hbase/hbase-protocol-shaded/target/generated-sources/protobuf/java/org/apache/hadoop/hbase/shaded/protobuf/generated/SnapshotProtos.java:[123,20] package javax.annotation does not exist [ERROR] /home/busbey/projects/hbase/hbase-protocol-shaded/target/generated-sources/protobuf/java/org/apache/hadoop/hbase/shaded/protobuf/generated/AccessControlProtos.java:[6,18] package javax.annotation does not exist ... (repeat for like 100 lines)
another example for the search engines:
[busbey@busbey-centos76 hbase]$ mvn -version Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T11:41:47-07:00) Maven home: /usr/local/share/applications/apache-maven-3.6.0 Java version: 11.0.1, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-11-openjdk-11.0.1.13-3.el7_6.x86_64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-957.el7.x86_64", arch: "amd64", family: "unix" [busbey@busbey-centos76 hbase]$ mvn -DskipTests --batch-mode install [INFO] Scanning for projects... ... [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/busbey/projects/hbase/hbase-it/src/test/java/org/apache/hadoop/hbase/RESTApiClusterManager.java:[39,25] package javax.xml.ws.http does not exist [ERROR] /home/busbey/projects/hbase/hbase-it/src/test/java/org/apache/hadoop/hbase/RESTApiClusterManager.java:[204,17] cannot find symbol symbol: class HTTPException location: class org.apache.hadoop.hbase.RESTApiClusterManager [ERROR] /home/busbey/projects/hbase/hbase-it/src/test/java/org/apache/hadoop/hbase/RESTApiClusterManager.java:[247,17] cannot find symbol symbol: class HTTPException location: class org.apache.hadoop.hbase.RESTApiClusterManager [INFO] 3 errors ...
So the goal here is just for compiling, not for running with JDK11?
I've tried this before and then I notice that I need to wait for hadoop to be compatible with JDK11 so I gave up... Lots of the problems are that we use lots of JavaEE related annotations and classes, which are removed in JDK11. And maybe the Unsafe stuffs are also a problem, maybe we could use the PlatformDependent class in our shaded netty...
yes just compiling. I've got it working locally and want to get things in place so I can automate keeping us ready for when Hadoop has a version that works for runtime.
also I have already run into the case where I was getting a new contributor set up and their use of homebrew meant that we broke right away. So if I've missed something and we can't compile with JDK11-targetting-JDK8 then we need a section in the contributor guide to specifically call out how to install jdk8. I would rather things just work.
I've got it working locally and want to get things in place so I can automate keeping us ready for when Hadoop has a version that works for runtime.
Great.
v0
- adds jdk profiles to add J2EE dependencies that were removed in JDK11
- makes the spotbugs stuff 1.8+ instead of 1.8 exactly
- can successfully pass mvn -DskipTests --batch-mode -e install assembly:single
- resultant java class files report they are JDK1.8 compatible
This would at least allow new contributors for most patches to get a build. Haven't tried any tests; I don't think anything with e.g. minicluster would go well.
The binary tarball that results from this assembly fails to launch a standalone cluster (running with JDK8). Haven't finished figuring out why yet. It's all problems for netty so I suspect something has messed up the lib directory.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 11s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-0 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
master Compile Tests | |||
0 | mvndep | 0m 14s | Maven dependency ordering for branch |
+1 | mvninstall | 4m 43s | master passed |
+1 | compile | 5m 55s | master passed |
+1 | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. |
+1 | javadoc | 3m 42s | master passed |
Patch Compile Tests | |||
0 | mvndep | 0m 14s | Maven dependency ordering for patch |
+1 | mvninstall | 4m 41s | the patch passed |
+1 | compile | 6m 7s | the patch passed |
+1 | javac | 6m 7s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 11s | The patch has no ill-formed XML file. |
+1 | shadedjars | 4m 35s | patch has no errors when building our shaded downstream artifacts. |
+1 | hadoopcheck | 9m 45s | Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0. |
+1 | javadoc | 3m 44s | the patch passed |
Other Tests | |||
+1 | unit | 221m 33s | root in the patch passed. |
+1 | asflicense | 3m 5s | The patch does not generate ASF License warnings. |
274m 13s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12956860/HBASE-21808.0.patch |
Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile |
uname | Linux 95ce87755296 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 GNU/Linux |
Build tool | maven |
Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh |
git revision | master / 16c7f5dac9 |
maven | version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
Default Java | 1.8.0_181 |
Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/15786/testReport/ |
Max. process+thread count | 5360 (vs. ulimit of 10000) |
modules | C: hbase-protocol-shaded hbase-protocol hbase-it hbase-shaded hbase-shaded/hbase-shaded-client hbase-assembly . U: . |
Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/15786/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
what do folks think about committing this as-is for an incremental improvement? I can make a jira to track "ensure the assembly built with JDK11-targetting-JDK8 can run". that way I can get a jenkins job going for trying out unit tests and see how much we can cover before I run out of time to work on this.
for reference I previously had standalone running on JDK8 working with the assembly built by JDK11-targetting-JDK8 so I do think I'll be able to close out that follow-on jira, I just haven't figured out yet what I didn't write down from then.
Tested this out locally and it works just fine (building HBase for jdk8 using jdk11). You have my +1
Looking again at https://hbase.apache.org/book.html#java – is it worth our adding another table for JDK compatibility: supported JDK for building vs. support JDK for running?
Looking again at https://hbase.apache.org/book.html#java – is it worth our adding another table for JDK compatibility: supported JDK for building vs. support JDK for running?
I don't think so. IIRC back when we were first adding jdk8 support we could run but not build and we just put that in a note on the section. I think a similar note would do here if we wanted to update on the status of jdk11.
SGTM. I didn't think it was necessary to add such an axis of compatibility. I think you're good here
Results for branch master
build #765 on builds.a.o: -1 overall
details (if available):
+1 general checks
– For more information see general report
-1 jdk8 hadoop2 checks
– For more information see jdk8 (hadoop2) report
-1 jdk8 hadoop3 checks
– For more information see jdk8 (hadoop3) report
+1 source release artifact
– See build output for details.
+1 client integration test
Results for branch branch-2
build #1654 on builds.a.o: -1 overall
details (if available):
+1 general checks
– For more information see general report
-1 jdk8 hadoop2 checks
– For more information see jdk8 (hadoop2) report
+1 jdk8 hadoop3 checks
– For more information see jdk8 (hadoop3) report
+1 source release artifact
– See build output for details.
+1 client integration test
IIRC back when we were first adding jdk8 support we could run but not build and we just put that in a note on the section
I did some testing with JDK 11 late last year and the whole stack (zookeeper, hadoop, hbase, and in our case phoenix) exhibited a surprising number of unit test failures at each layer. Zookeeper results were particularly surprising, and concerning. I would hesitate to claim everything just works given test failures at every layer, but that could be the case. There is just no way to verify it.
Would this change be useful for branch-1 too?
Would this change be useful for branch-1 too?
it probably would be useful in branch-1; that was my original plan. I can't recall what issue I ran into when I tried but it was enough that I moved on.
IIRC back when we were first adding jdk8 support we could run but not build and we just put that in a note on the section
I did some testing with JDK 11 late last year and the whole stack (zookeeper, hadoop, hbase, and in our case phoenix) exhibited a surprising number of unit test failures at each layer. Zookeeper results were particularly surprising, and concerning. I would hesitate to claim everything just works given test failures at every layer, but that could be the case. There is just no way to verify it.
Oh sorry if I wasn't clear. IIRC we specifically went through the integration tests including ITBLL when running on jdk8 after building with jdk7. I didn't mean to imply that "if it starts it ships" was the bar.
it probably would be useful in branch-1; that was my original plan. I can't recall what issue I ran into when I tried but it was enough that I moved on.
Ok then let's wait until there is a real itch before we spend resources trying to scratch it
Looks like the Rest server is broke after this. {java.lang.NoClassDefFoundError: javax/annotation/Priority}.
ooof. Okay I haven't dug in, but the bits of reading I've done seem to point to a javax-annotations change for jdk8 where the version went from 1.1 -> 1.2 and the Priority class got dropped.
Can you open a jira to confirm and then track adding just the needed dependency to keep the correct version of the annotations library around? (if there's some dependency of the rest server we could upgrade to make this problem go away on jdk8 I'd like that even more. )
Edited the description to remove RHEL7.6, as my retesting of assumptions got me jdk1.8