Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Java 8 is coming quickly to various clusters. Making sure Hadoop seamlessly works with Java 8 is important for the Apache community.

      This JIRA is to track the issues/experiences encountered during Java 8 migration. If you find a potential bug , please create a separate JIRA either as a sub-task or linked into this JIRA.
      If you find a Hadoop or JVM configuration tuning, you can create a JIRA as well. Or you can add a comment here.

        Issue Links

          Activity

          Hide
          apurtell Andrew Purtell added a comment -

          You'll have issues with Javadoc lint during compile. It can be disabled by a javac flag, unfortunately not recognized by javac < 8, but maybe some Maven magic could take care of that. Or, the javadoc itself can be fixed.

          Show
          apurtell Andrew Purtell added a comment - You'll have issues with Javadoc lint during compile . It can be disabled by a javac flag, unfortunately not recognized by javac < 8, but maybe some Maven magic could take care of that. Or, the javadoc itself can be fixed.
          Hide
          apurtell Andrew Purtell added a comment -

          See HBASE-12006, we borrowed this class from core test.

          Show
          apurtell Andrew Purtell added a comment - See HBASE-12006 , we borrowed this class from core test.
          Hide
          acmurthy Arun C Murthy added a comment -

          I'd like to try get this into 2.7 if possible, thoughts?

          Thanks for driving this Mohammad Kamrul Islam!

          Show
          acmurthy Arun C Murthy added a comment - I'd like to try get this into 2.7 if possible, thoughts? Thanks for driving this Mohammad Kamrul Islam !
          Hide
          kamrul Mohammad Kamrul Islam added a comment -

          Yes.

          We are running everything with Java 8. As it is running, we want to find out the issues or better configurations.
          We will post the findings as we go.

          In short, things are ok except : heap and VM usages are little higher in some instances.Our team is working to get handle of this.

          Show
          kamrul Mohammad Kamrul Islam added a comment - Yes. We are running everything with Java 8. As it is running, we want to find out the issues or better configurations. We will post the findings as we go. In short, things are ok except : heap and VM usages are little higher in some instances.Our team is working to get handle of this.
          Hide
          airbots Chen He added a comment -

          Hi Andrew Purtell, I post the patch that can disable doclint when you create hadoop package. Don't know whether it is helpful for you or not.

          Show
          airbots Chen He added a comment - Hi Andrew Purtell , I post the patch that can disable doclint when you create hadoop package. Don't know whether it is helpful for you or not.
          Hide
          kamrul Mohammad Kamrul Islam added a comment -

          I took some other short-cut to build with Java 8 by disabling the java-doc generation.
          I passed -Dmaven.javadoc.skip=true in the mvn command.

          However, we must resolve this either by disabling doclint or by changing the doc manually.

          Show
          kamrul Mohammad Kamrul Islam added a comment - I took some other short-cut to build with Java 8 by disabling the java-doc generation. I passed -Dmaven.javadoc.skip=true in the mvn command. However, we must resolve this either by disabling doclint or by changing the doc manually.
          Hide
          apurtell Andrew Purtell added a comment -

          I post the patch that can disable doclint when you create hadoop package.

          Does it work when you use Java 7 to build?

          Show
          apurtell Andrew Purtell added a comment - I post the patch that can disable doclint when you create hadoop package. Does it work when you use Java 7 to build?
          Hide
          airbots Chen He added a comment -

          Hi Andrew Purtell, I just tested Hadoop-11292.patch using Java.1.7.0_71 against trunk. Running "mvn clean package -DskipTests" and "mvn clean install -DskipTests" both succeed.

          Show
          airbots Chen He added a comment - Hi Andrew Purtell , I just tested Hadoop-11292.patch using Java.1.7.0_71 against trunk. Running "mvn clean package -DskipTests" and "mvn clean install -DskipTests" both succeed.
          Hide
          puce Florian Brunner added a comment -

          Please note: Oracle Java SE 7 is planned going EoL on April 2015, so in 3-4 months from now: http://www.oracle.com/technetwork/java/eol-135779.html

          It would be great if Hadoop would work well with Java SE 8 by then, since developers are encouraged to update to Java SE 8 if they don't have Oracle support contracts.

          What is the planned release date of v2.7.0?

          Show
          puce Florian Brunner added a comment - Please note: Oracle Java SE 7 is planned going EoL on April 2015, so in 3-4 months from now: http://www.oracle.com/technetwork/java/eol-135779.html It would be great if Hadoop would work well with Java SE 8 by then, since developers are encouraged to update to Java SE 8 if they don't have Oracle support contracts. What is the planned release date of v2.7.0?
          Hide
          rkellogg Rick Kellogg added a comment -

          Suggest we look into using Animal Sniffer (http://mojo.codehaus.org/animal-sniffer/) to ensure API compatibility between releases.

          The folks at Spring Source use it to help with Java 6-8 compatibility as well.

          See: http://spring.io/blog/2015/04/03/how-spring-achieves-compatibility-with-java-6-7-and-8

          Show
          rkellogg Rick Kellogg added a comment - Suggest we look into using Animal Sniffer ( http://mojo.codehaus.org/animal-sniffer/ ) to ensure API compatibility between releases. The folks at Spring Source use it to help with Java 6-8 compatibility as well. See: http://spring.io/blog/2015/04/03/how-spring-achieves-compatibility-with-java-6-7-and-8
          Hide
          dpinol Daniel Pinyol added a comment -

          Any release date for this?

          Show
          dpinol Daniel Pinyol added a comment - Any release date for this?
          Hide
          andrew.wang Andrew Wang added a comment -

          We could probably close this JIRA. branch-2 seems to run fine on JDK8, and we've been fixing up compilation issues as we find them.

          I'm hoping to bump the JDK min version to JDK8 in trunk pretty soon, since as noted earlier JDK7 is EOL this month.

          Show
          andrew.wang Andrew Wang added a comment - We could probably close this JIRA. branch-2 seems to run fine on JDK8, and we've been fixing up compilation issues as we find them. I'm hoping to bump the JDK min version to JDK8 in trunk pretty soon, since as noted earlier JDK7 is EOL this month.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I'm hoping to bump the JDK min version to JDK8 in trunk pretty soon, since as noted earlier JDK7 is EOL this month.

          Hi Andrew, it should probably be put to vote (unless the community already agreed to it and I missed it). I assume branch-2 remains on Java 7 for the foreseeable future.

          Thanks, Arpit.

          Show
          arpitagarwal Arpit Agarwal added a comment - I'm hoping to bump the JDK min version to JDK8 in trunk pretty soon, since as noted earlier JDK7 is EOL this month. Hi Andrew, it should probably be put to vote (unless the community already agreed to it and I missed it). I assume branch-2 remains on Java 7 for the foreseeable future. Thanks, Arpit.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          In fact, I think 2.7 is ready for JDK 7 since almost works are done:

          Anyway, let's vote about this topic.

          Show
          ozawa Tsuyoshi Ozawa added a comment - In fact, I think 2.7 is ready for JDK 7 since almost works are done: HADOOP-11098 , HADOOP-11364 are configuration problems. MAPREDUCE-6165 is only test problem. It's not related to body of Hadoop. Anyway, let's vote about this topic.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Oh, sorry, I meant 2.7 is ready for JDK 8.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Oh, sorry, I meant 2.7 is ready for JDK 8.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          s/JDK 7/JDK 8/

          Show
          ozawa Tsuyoshi Ozawa added a comment - s/JDK 7/JDK 8/
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          In fact, I think 2.7 is ready for JDK 7 since almost works are done:

          I see, I thought branch-2 cannot be forced to require JDK 8 for compatibility reasons however it might be permitted by our compatibility guide.

          The JVM requirements will not change across point releases within the same minor release except if the JVM version under question becomes unsupported. Minor/major releases might require later versions of JVM for some/all of the supported operating systems.

          Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - In fact, I think 2.7 is ready for JDK 7 since almost works are done: I see, I thought branch-2 cannot be forced to require JDK 8 for compatibility reasons however it might be permitted by our compatibility guide. The JVM requirements will not change across point releases within the same minor release except if the JVM version under question becomes unsupported. Minor/major releases might require later versions of JVM for some/all of the supported operating systems. Thanks.
          Hide
          alxt Alexey Tomin added a comment -

          "The upcoming release of Java 7 update 80 (April 2015) marks the last public release in Orale’s JDK 7 family."
          Hadoop is NOT JDK8 ready

          Show
          alxt Alexey Tomin added a comment - "The upcoming release of Java 7 update 80 (April 2015) marks the last public release in Orale’s JDK 7 family." Hadoop is NOT JDK8 ready
          Hide
          stevel@apache.org Steve Loughran added a comment -

          2.6 doesn't work on a secure JDK8 cluster. Have you seen any problems with the 2.7.0 RC0?

          Show
          stevel@apache.org Steve Loughran added a comment - 2.6 doesn't work on a secure JDK8 cluster. Have you seen any problems with the 2.7.0 RC0?
          Hide
          steveapache Steve added a comment -

          Commenting on the possible severity of this issue –

          JDK7 has been discontinued by Oracle, yet the last publicly available release now has a day-zero vulnerability "actively being exploited by hackers".

          See:
          https://blogs.oracle.com/security/entry/july_2015_critical_patch_update
          http://www.oracle.com/technetwork/topics/security/cpujul2015-2367936.html

          Show
          steveapache Steve added a comment - Commenting on the possible severity of this issue – JDK7 has been discontinued by Oracle, yet the last publicly available release now has a day-zero vulnerability "actively being exploited by hackers". See: https://blogs.oracle.com/security/entry/july_2015_critical_patch_update http://www.oracle.com/technetwork/topics/security/cpujul2015-2367936.html
          Hide
          rkanter Robert Kanter added a comment -

          The nightly JDK8 builds are definitely in a better state than they were before. Common has been mostly green now, MR was until yesterday, and YARN has been mostly green lately too. HDFS's build still needs some work though.

          That said, I'm not sure it makes sense to wait for all of these builds to be green before switching over. With the number of commits going into Hadoop each day, we're basically trying to hit a moving target here. I think we might be better of switching the pre-commit build to JDK8 so that we can catch anything before it goes in. And I guess we should probably get YARN-3528 moving again as well.

          Show
          rkanter Robert Kanter added a comment - The nightly JDK8 builds are definitely in a better state than they were before. Common has been mostly green now, MR was until yesterday, and YARN has been mostly green lately too. HDFS's build still needs some work though. That said, I'm not sure it makes sense to wait for all of these builds to be green before switching over. With the number of commits going into Hadoop each day, we're basically trying to hit a moving target here. I think we might be better of switching the pre-commit build to JDK8 so that we can catch anything before it goes in. And I guess we should probably get YARN-3528 moving again as well.
          Hide
          aw Allen Wittenauer added a comment -

          I think we might be better of switching the pre-commit build to JDK8 so that we can catch anything before it goes in.

          Yetus supports testing against multiple JDKs during precommit runs. But in order to make it even remotely viable for Hadoop, two things need to happen:

          • HADOOP-12198 needs to get committed to Yetus to fix Hadoop support back up
          • HADOOP-11984 needs to get committed to Hadoop so that unit tests don't take 4 hours
          Show
          aw Allen Wittenauer added a comment - I think we might be better of switching the pre-commit build to JDK8 so that we can catch anything before it goes in. Yetus supports testing against multiple JDKs during precommit runs. But in order to make it even remotely viable for Hadoop, two things need to happen: HADOOP-12198 needs to get committed to Yetus to fix Hadoop support back up HADOOP-11984 needs to get committed to Hadoop so that unit tests don't take 4 hours
          Hide
          stevel@apache.org Steve Loughran added a comment -

          link to HADOOP-11628: SPNEGO auth does not work with CNAMEs in JDK8

          Show
          stevel@apache.org Steve Loughran added a comment - link to HADOOP-11628 : SPNEGO auth does not work with CNAMEs in JDK8
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Now, the source code can be compiled with JDK8 after YARN-4294. It's time to take a look at issues about dependencies - HADOOP-11993(maven-enforcer-plugin), HADOOP-12064(guice), HADOOP-9613(asm and jersey).

          Show
          ozawa Tsuyoshi Ozawa added a comment - Now, the source code can be compiled with JDK8 after YARN-4294 . It's time to take a look at issues about dependencies - HADOOP-11993 (maven-enforcer-plugin), HADOOP-12064 (guice), HADOOP-9613 (asm and jersey).
          Hide
          tsuna Benoit Sigoure added a comment -

          Is this JIRA still relevant? I've also been running Hadoop+HBase on JDK8 just fine for quite some time now, but curious to know whether there are any known outstanding issues.

          Show
          tsuna Benoit Sigoure added a comment - Is this JIRA still relevant? I've also been running Hadoop+HBase on JDK8 just fine for quite some time now, but curious to know whether there are any known outstanding issues.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          it is, as its the corner cases. For example: were you on kerberos?

          Show
          stevel@apache.org Steve Loughran added a comment - it is, as its the corner cases. For example: were you on kerberos?
          Hide
          tsuna Benoit Sigoure added a comment -

          Nope. What else is known to break?

          Show
          tsuna Benoit Sigoure added a comment - Nope. What else is known to break?
          Hide
          andrew.wang Andrew Wang added a comment -

          I think we're safe to resolve this JIRA.

          CDH blacklists a few versions, but certified with JDK8 (based on heavily modified 2.6):

          https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_jdk

          HDP seems similar (2.7 based):

          http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_Installing_HDP_AMB/content/_jdk_requirements.html

          We also bumped the required JDK version to JDK8 for 3.0.0-alpha1.

          If there are additional JDK8 issues, let's follow up with separate JIRAs. Thanks all.

          Show
          andrew.wang Andrew Wang added a comment - I think we're safe to resolve this JIRA. CDH blacklists a few versions, but certified with JDK8 (based on heavily modified 2.6): https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_jdk HDP seems similar (2.7 based): http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_Installing_HDP_AMB/content/_jdk_requirements.html We also bumped the required JDK version to JDK8 for 3.0.0-alpha1. If there are additional JDK8 issues, let's follow up with separate JIRAs. Thanks all.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks for this...now it's time to see if we can use Java 9 to deal with classpath hell downstream...

          Show
          stevel@apache.org Steve Loughran added a comment - thanks for this...now it's time to see if we can use Java 9 to deal with classpath hell downstream...
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          I'm very happy to see that this ticket is closed! Thanks all people for helping!

          Show
          ozawa Tsuyoshi Ozawa added a comment - I'm very happy to see that this ticket is closed! Thanks all people for helping!

            People

            • Assignee:
              kamrul Mohammad Kamrul Islam
              Reporter:
              kamrul Mohammad Kamrul Islam
            • Votes:
              20 Vote for this issue
              Watchers:
              104 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development