Details

    • Type: Umbrella
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.1.0, 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Java 9 will ship in 2016. This will be the first Java release that makes a significant compatibility departure from earlier runtimes.

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          ugh. I think we need a DISCUSS thread on Java strategies. Oracle announce EOL for Java 9 is going to be March 2018 which is like super soon and before the end of Java 8. The next LTS release won't be GA until September 2018.

          Show
          busbey Sean Busbey added a comment - ugh. I think we need a DISCUSS thread on Java strategies. Oracle announce EOL for Java 9 is going to be March 2018 which is like super soon and before the end of Java 8. The next LTS release won't be GA until September 2018.
          Hide
          apurtell Andrew Purtell added a comment -

          Making a subtask to increase the Hadoop version for branch-1 and branch-1.4 to 2.7.4 (from 2.5.1)

          Show
          apurtell Andrew Purtell added a comment - Making a subtask to increase the Hadoop version for branch-1 and branch-1.4 to 2.7.4 (from 2.5.1)
          Hide
          apurtell Andrew Purtell added a comment -

          Not a blocker

          Show
          apurtell Andrew Purtell added a comment - Not a blocker
          Hide
          busbey Sean Busbey added a comment -

          so subtask to upgrade the shipped-with version in master to Hadoop 2.7.4 and we can add a note in one or both of the hadoop or java sections about the requirement?

          Show
          busbey Sean Busbey added a comment - so subtask to upgrade the shipped-with version in master to Hadoop 2.7.4 and we can add a note in one or both of the hadoop or java sections about the requirement?
          Hide
          mdrob Mike Drob added a comment - - edited

          Tried to start a java8 compiled hbase in a java9 env:

          mdrob@mdrob-MBP:/tmp/hb2a4/hbase-2.0.0-alpha4$ ./bin/start-hbase.sh
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
          Exception in thread "main" java.lang.ExceptionInInitializerError
          	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
          	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
          	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66)
          	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80)
          	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94)
          	at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)
          Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1
          	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116)
          	at java.base/java.lang.String.substring(String.java:1885)
          	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:51)
          	... 6 more
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
          Exception in thread "main" java.lang.ExceptionInInitializerError
          	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
          	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
          	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66)
          	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80)
          	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94)
          	at org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:60)
          Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1
          	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116)
          	at java.base/java.lang.String.substring(String.java:1885)
          	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:51)
          	... 6 more
          running master, logging to /tmp/hb2a4/hbase-2.0.0-alpha4/bin/../logs/hbase-mdrob-master-mdrob-MBP.local.out
          Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
          Exception in thread "main" java.lang.ExceptionInInitializerError
          	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
          	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
          	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66)
          	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80)
          	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94)
          	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
          	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2848)
          Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1
          : running regionserver, logging to /tmp/hb2a4/hbase-2.0.0-alpha4/bin/../logs/hbase-mdrob-regionserver-mdrob-MBP.local.out
          : Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
          : Exception in thread "main" java.lang.ExceptionInInitializerError
          : 	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
          : 	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
          : 	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66)
          : 	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80)
          : 	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94)
          : 	at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3029)
          : Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1
          : 	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116)
          

          Looks like HADOOP-14586.

          Show
          mdrob Mike Drob added a comment - - edited Tried to start a java8 compiled hbase in a java9 env: mdrob@mdrob-MBP:/tmp/hb2a4/hbase-2.0.0-alpha4$ ./bin/start-hbase.sh Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94) at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39) Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1 at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116) at java.base/java.lang.String.substring(String.java:1885) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:51) ... 6 more Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94) at org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:60) Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1 at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116) at java.base/java.lang.String.substring(String.java:1885) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:51) ... 6 more running master, logging to /tmp/hb2a4/hbase-2.0.0-alpha4/bin/../logs/hbase-mdrob-master-mdrob-MBP.local.out Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2848) Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1 : running regionserver, logging to /tmp/hb2a4/hbase-2.0.0-alpha4/bin/../logs/hbase-mdrob-regionserver-mdrob-MBP.local.out : Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. : Exception in thread "main" java.lang.ExceptionInInitializerError : at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) : at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437) : at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:66) : at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:80) : at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:94) : at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3029) : Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 1 : at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3116) Looks like HADOOP-14586 .
          Hide
          busbey Sean Busbey added a comment -

          I think we should kick this out and do it in the HBase 2.1.0 release.

          Show
          busbey Sean Busbey added a comment - I think we should kick this out and do it in the HBase 2.1.0 release.
          Hide
          busbey Sean Busbey added a comment -

          I'm on OSX and the download I got from the jdk 9 site doesn't look like a beta:

          $ /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/bin/java -version
          java version "9"
          Java(TM) SE Runtime Environment (build 9+181)
          Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

          For comparison, my last beta install was named like jdk1.9-ea-b102.jdk

          Show
          busbey Sean Busbey added a comment - I'm on OSX and the download I got from the jdk 9 site doesn't look like a beta: $ /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/bin/java -version java version "9" Java(TM) SE Runtime Environment (build 9+181) Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode) For comparison, my last beta install was named like jdk1.9-ea-b102.jdk
          Hide
          mdrob Mike Drob added a comment -

          On the downloads page, it claims that "Builds for platforms other than Linux/x64 will be published at a later date."

          So those of us developing on OSX are still stuck with beta builds.

          Show
          mdrob Mike Drob added a comment - On the downloads page, it claims that "Builds for platforms other than Linux/x64 will be published at a later date." So those of us developing on OSX are still stuck with beta builds.
          Hide
          busbey Sean Busbey added a comment -

          Maybe we should pull this back into one of the beta releases; it sounds like the changes it requires might have some perf impact on jdk8 folks?

          Show
          busbey Sean Busbey added a comment - Maybe we should pull this back into one of the beta releases; it sounds like the changes it requires might have some perf impact on jdk8 folks?
          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -

          Findbugs does not work and the comment above also needs to be fixed
          https://issues.apache.org/jira/browse/HBASE-12081?focusedCommentId=16036746&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16036746
          In case you need to make the server run.

          Show
          ram_krish ramkrishna.s.vasudevan added a comment - Findbugs does not work and the comment above also needs to be fixed https://issues.apache.org/jira/browse/HBASE-12081?focusedCommentId=16036746&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16036746 In case you need to make the server run.
          Hide
          stack stack added a comment -

          Thanks Mike Drob

          Show
          stack stack added a comment - Thanks Mike Drob
          Hide
          mdrob Mike Drob added a comment -

          I had looked at this before and the first big blockers that I found were that the version of findbugs we use was not compatible (I think because the underlying asm needs to be re-instrumented for Java 9) and that scala needed an update (so hbase-spark has issues). Commenting those bits out, we could at least compile, but I never finished getting tests to pass.

          Show
          mdrob Mike Drob added a comment - I had looked at this before and the first big blockers that I found were that the version of findbugs we use was not compatible (I think because the underlying asm needs to be re-instrumented for Java 9) and that scala needed an update (so hbase-spark has issues). Commenting those bits out, we could at least compile, but I never finished getting tests to pass.
          Hide
          apurtell Andrew Purtell added a comment -

          I also added branch-1 and branch-1.4, but I fear that may be optimistic. Will drop branch-1.4 if need be. We can keep this as a blocker for continuing forward with branch-1 (1.5, etc.)

          Show
          apurtell Andrew Purtell added a comment - I also added branch-1 and branch-1.4, but I fear that may be optimistic. Will drop branch-1.4 if need be. We can keep this as a blocker for continuing forward with branch-1 (1.5, etc.)
          Hide
          stack stack added a comment -

          Making this a blocker for 2.0.0. Ensure we at least run on a jdk9.

          Show
          stack stack added a comment - Making this a blocker for 2.0.0. Ensure we at least run on a jdk9.
          Hide
          davelatham Dave Latham added a comment -

          Java 9 is now GA

          Show
          davelatham Dave Latham added a comment - Java 9 is now GA
          Hide
          mdrob Mike Drob added a comment -

          FYI: jdk-9+175, the first java 9 RC is now available at http://jdk.java.net/9/

          Show
          mdrob Mike Drob added a comment - FYI: jdk-9+175, the first java 9 RC is now available at http://jdk.java.net/9/
          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -

          Considering the fact that we are accessing the private members of sun.nio, java.lang.reflect etc. All those will fail. Any code using setAccessible(true) will fail to work. Java 9 suggests some alternate for this (not sure whether we will have a better work around when the actual release happens).
          Alternate includes adding these extns to JVM
          (for eg)

          --add-exports=java.base/sun.nio.ch=ALL-UNNAMED
          --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
          

          Just adding it here as I could see this JIRA as the umbrella JIRA.

          Show
          ram_krish ramkrishna.s.vasudevan added a comment - Considering the fact that we are accessing the private members of sun.nio, java.lang.reflect etc. All those will fail. Any code using setAccessible(true) will fail to work. Java 9 suggests some alternate for this (not sure whether we will have a better work around when the actual release happens). Alternate includes adding these extns to JVM (for eg) --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED Just adding it here as I could see this JIRA as the umbrella JIRA.
          Hide
          apurtell Andrew Purtell added a comment -

          For an earlier discussion with folks at Oracle I ran jdeps analysis over the core Hadoop ecosystem and related projects that have notable or emerging user bases (in my opinion, some may debate...): Hadoop, ZooKeeper, Hive(+Tez), Pig, HBase, Oozie, Flume, Sqoop, Avro, Thrift, Accumulo, Cassandra, Crunch, Cascading, Elasticsearch, Giraph, Kafka, Mahout, Parquet, Optiq, Solr/Lucene, Spark, Storm. I picked recent releases except for HBase and Optiq, which I built from source. For Optiq because I could not find a binary distribution at the time. For HBase I used 0.99-SNAPSHOT because it has the new dependencies on Netty 4 and the LMAX Disruptor that will be in the upcoming 1.0 release. There are some common dependencies (Netty, Guava, Derby, the Scala runtime, Coda Hale's Metrics, LMAX Disruptor) that use internal sun.* APIs. We would need versions of these dependencies which do not make those calls available before we could eliminate internal API calls through this route. Other uses fall into a few common patterns. Daemons use Signal/SignalHandler for catching signals. Datastores such as HBase, Hive, and Cassandra are seeking their own high performance optimizations with Unsafe, including direct memory allocation. Avro and Kyro are serialization libraries seeking high performance via Unsafe on hot code paths. Hadoop and HBase utilize some security APIs for Kerberos and SSL which seem to have no non-internal API analogues. Pig and HBase pull in Jython and JRuby, respectively. Jython uses a bit of Unsafe. JRuby is an aggressive user of JVM internals.

          Show
          apurtell Andrew Purtell added a comment - For an earlier discussion with folks at Oracle I ran jdeps analysis over the core Hadoop ecosystem and related projects that have notable or emerging user bases (in my opinion, some may debate...): Hadoop, ZooKeeper, Hive(+Tez), Pig, HBase, Oozie, Flume, Sqoop, Avro, Thrift, Accumulo, Cassandra, Crunch, Cascading, Elasticsearch, Giraph, Kafka, Mahout, Parquet, Optiq, Solr/Lucene, Spark, Storm. I picked recent releases except for HBase and Optiq, which I built from source. For Optiq because I could not find a binary distribution at the time. For HBase I used 0.99-SNAPSHOT because it has the new dependencies on Netty 4 and the LMAX Disruptor that will be in the upcoming 1.0 release. There are some common dependencies (Netty, Guava, Derby, the Scala runtime, Coda Hale's Metrics, LMAX Disruptor) that use internal sun.* APIs. We would need versions of these dependencies which do not make those calls available before we could eliminate internal API calls through this route. Other uses fall into a few common patterns. Daemons use Signal/SignalHandler for catching signals. Datastores such as HBase, Hive, and Cassandra are seeking their own high performance optimizations with Unsafe, including direct memory allocation. Avro and Kyro are serialization libraries seeking high performance via Unsafe on hot code paths. Hadoop and HBase utilize some security APIs for Kerberos and SSL which seem to have no non-internal API analogues. Pig and HBase pull in Jython and JRuby, respectively. Jython uses a bit of Unsafe. JRuby is an aggressive user of JVM internals.
          Hide
          esteban Esteban Gutierrez added a comment -

          Right, I ran the same tests a moment ago and we also have some dependencies, including Cliff Click's high perf lib that rely on sun.misc.Unsafe also JRuby falls into that category.

          Show
          esteban Esteban Gutierrez added a comment - Right, I ran the same tests a moment ago and we also have some dependencies, including Cliff Click's high perf lib that rely on sun.misc.Unsafe also JRuby falls into that category.
          Hide
          apurtell Andrew Purtell added a comment - - edited

          A number of "internal" com.sun classes will be removed, including Unsafe. These removals need to be handled or HBase will not be able to run on a Java 9 runtime. Doing this in a backwards compatible way may be tricky. I'm not sure how the Java platform community will navigate to where they want to go while dealing with unfixable user code breakage and I hope we have an opportunity to provide pushback on changes if required.

          I don't think we need to do anything today beyond track what others in the ecosystem are doing, for example HADOOP-11123

          Show
          apurtell Andrew Purtell added a comment - - edited A number of "internal" com.sun classes will be removed, including Unsafe. These removals need to be handled or HBase will not be able to run on a Java 9 runtime. Doing this in a backwards compatible way may be tricky. I'm not sure how the Java platform community will navigate to where they want to go while dealing with unfixable user code breakage and I hope we have an opportunity to provide pushback on changes if required. I don't think we need to do anything today beyond track what others in the ecosystem are doing, for example HADOOP-11123

            People

            • Assignee:
              busbey Sean Busbey
              Reporter:
              apurtell Andrew Purtell
            • Votes:
              2 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:

                Development