Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11858

[JDK8] Set minimum version of Hadoop 3 to JDK 8

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The minimum required JDK version for Hadoop has been increased from JDK7 to JDK8.

      Description

      Set minimum version of trunk to JDK 8

      1. commit.ogv
        1.28 MB
        Andrew Wang
      2. HADOOP-11858.001.patch
        0.9 kB
        Robert Kanter
      3. HADOOP-11858.002.patch
        2 kB
        Robert Kanter
      4. HADOOP-11858.003.patch
        3 kB
        Andrew Wang

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          One comment, we should update the BUILDING.txt instructions too. Otherwise +1.

          We of course also need to do the related Jenkins-side work, which is where the real fun lies.

          Show
          andrew.wang Andrew Wang added a comment - One comment, we should update the BUILDING.txt instructions too. Otherwise +1. We of course also need to do the related Jenkins-side work, which is where the real fun lies.
          Hide
          rkanter Robert Kanter added a comment -

          002 patch also updates BUILDING.txt

          Show
          rkanter Robert Kanter added a comment - 002 patch also updates BUILDING.txt
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Actually I didn't see any conclusion on this on the previous thread on the dev lists. There were proposals and counter points, but we never made a decision. We should get consensus first, given this has implications on how branch-2 evolves. I propose we reignite this on the dev lists before moving ahead.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Actually I didn't see any conclusion on this on the previous thread on the dev lists. There were proposals and counter points, but we never made a decision. We should get consensus first, given this has implications on how branch-2 evolves. I propose we reignite this on the dev lists before moving ahead.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Given the effective timetable of any 3.x release, I think it should be jdk8; though clarifying this on the dev list one more time may be wise.

          1. We need a longer time before forcing java7 on applications using Hadoop; a split from client to server libs where client is @ 7 would address that, albeit at a more complex build.
          2. Based on my experience with the Java 7 move, before applying this patch, it is absolutely critical that jenkins is working on Java 8. without the tests being stable first, it's impossible to be sure that the move took place properly.
          3. of course, the Jenkins change here would essentially be "kill the trunk & java7 builds; rename the java 8 one"

          I just got a patch for one problem in there yesterday (HADOOP-11846), but now it looks like there's another, HADOOP-11864

          YARN on Java 8 is broken simply because someone thought it was a good idea to use 12345 as the port for all test runs, so breaking on parallel runs. That needs to be fixed to hide false alarms. I'm not going to do the code for that myself, but I promise to +1 a patch which does the port-scanning adequately.

          Show
          stevel@apache.org Steve Loughran added a comment - Given the effective timetable of any 3.x release, I think it should be jdk8; though clarifying this on the dev list one more time may be wise. We need a longer time before forcing java7 on applications using Hadoop; a split from client to server libs where client is @ 7 would address that, albeit at a more complex build. Based on my experience with the Java 7 move, before applying this patch, it is absolutely critical that jenkins is working on Java 8. without the tests being stable first, it's impossible to be sure that the move took place properly. of course, the Jenkins change here would essentially be "kill the trunk & java7 builds; rename the java 8 one" I just got a patch for one problem in there yesterday ( HADOOP-11846 ), but now it looks like there's another, HADOOP-11864 YARN on Java 8 is broken simply because someone thought it was a good idea to use 12345 as the port for all test runs, so breaking on parallel runs. That needs to be fixed to hide false alarms. I'm not going to do the code for that myself, but I promise to +1 a patch which does the port-scanning adequately.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          YARN-3433 : hard coded jetty ports

          Show
          stevel@apache.org Steve Loughran added a comment - YARN-3433 : hard coded jetty ports
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Link to YARN-3528 Tests with 12345 as hard-coded port break jenkins

          Show
          stevel@apache.org Steve Loughran added a comment - Link to YARN-3528 Tests with 12345 as hard-coded port break jenkins
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Can we remove -XX:MaxPermSize setting from pom.xml in this issue? The option is not supported in JDK8. Please see HADOOP-12571 for the detail.

          Show
          ajisakaa Akira Ajisaka added a comment - Can we remove -XX:MaxPermSize setting from pom.xml in this issue? The option is not supported in JDK8. Please see HADOOP-12571 for the detail.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Given its 2016, is it time to raise this topic on the dev@ list again?

          Show
          stevel@apache.org Steve Loughran added a comment - Given its 2016, is it time to raise this topic on the dev@ list again?
          Hide
          andrew.wang Andrew Wang added a comment -

          I'm still +1 for bumping.

          Show
          andrew.wang Andrew Wang added a comment - I'm still +1 for bumping.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Given its 2016, is it time to raise this topic on the dev@ list again?

          I think yes. As Vinod said, we should get consensus first.

          Show
          ajisakaa Akira Ajisaka added a comment - Given its 2016, is it time to raise this topic on the dev@ list again? I think yes. As Vinod said, we should get consensus first.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Raised this topic to the dev@ lists.

          Show
          ajisakaa Akira Ajisaka added a comment - Raised this topic to the dev@ lists.
          Hide
          andrew.wang Andrew Wang added a comment -

          dev list seemed to agree, anything left besides committing this patch? If not, I plan to commit EOD tomorrow.

          Show
          andrew.wang Andrew Wang added a comment - dev list seemed to agree, anything left besides committing this patch? If not, I plan to commit EOD tomorrow.
          Hide
          rkanter Robert Kanter added a comment -

          Despite the patch being ~1 year old, it still seems to apply and work, so that's neat
          Thanks everyone for pushing this forward; it'll be nice to get this in after all this time.

          Show
          rkanter Robert Kanter added a comment - Despite the patch being ~1 year old, it still seems to apply and work, so that's neat Thanks everyone for pushing this forward; it'll be nice to get this in after all this time.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          anything left besides committing this patch?

          As I commented previously, can we remove -XX:MaxPermSize setting from pom.xml? The option is not supported in JDK8.

          Show
          ajisakaa Akira Ajisaka added a comment - anything left besides committing this patch? As I commented previously, can we remove -XX:MaxPermSize setting from pom.xml? The option is not supported in JDK8.
          Hide
          andrew.wang Andrew Wang added a comment -

          Here's a patch getting rid of the MaxPermSize flag. I tested by doing a full build with this patch; enforcer rule triggered correctly.

          Show
          andrew.wang Andrew Wang added a comment - Here's a patch getting rid of the MaxPermSize flag. I tested by doing a full build with this patch; enforcer rule triggered correctly.
          Hide
          andrew.wang Andrew Wang added a comment -

          Not sure if we can really run this through pre-commit since we can't run full test runs anymore, and all we're really looking for is validation that it builds.

          If someone else +1's, I'll go ahead and commit (full credit to Robert).

          Show
          andrew.wang Andrew Wang added a comment - Not sure if we can really run this through pre-commit since we can't run full test runs anymore, and all we're really looking for is validation that it builds. If someone else +1's, I'll go ahead and commit (full credit to Robert).
          Hide
          rkanter Robert Kanter added a comment -

          I've confirmed that with the 003 patch, the maxPermSize warning goes away and everything builds with Java 8, but the enforcer blocks Java 7 and 6. So +1 from me, but I don't know if you want to wait for a less "biased" +1 from someone else

          Show
          rkanter Robert Kanter added a comment - I've confirmed that with the 003 patch, the maxPermSize warning goes away and everything builds with Java 8, but the enforcer blocks Java 7 and 6. So +1 from me, but I don't know if you want to wait for a less "biased" +1 from someone else
          Hide
          andrew.wang Andrew Wang added a comment -

          I think we can use my +1 for 002 and your +1 for the 003 diff.

          Thanks Robert, I'll commit this shortly.

          Show
          andrew.wang Andrew Wang added a comment - I think we can use my +1 for 002 and your +1 for the 003 diff. Thanks Robert, I'll commit this shortly.
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk. Thanks Robert for the patch and everyone for reviewing!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk. Thanks Robert for the patch and everyone for reviewing!
          Hide
          andrew.wang Andrew Wang added a comment -

          Per Steve's request on the dev list, I'm also attaching a video of this patch being committed for posterity.

          Show
          andrew.wang Andrew Wang added a comment - Per Steve's request on the dev list, I'm also attaching a video of this patch being committed for posterity.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9770 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9770/)
          HADOOP-11858. [JDK8] Set minimum version of Hadoop 3 to JDK 8. (wang: rev 4b55642b9d836691592405805c181d12c2ed7e50)

          • hadoop-project/pom.xml
          • pom.xml
          • BUILDING.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9770 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9770/ ) HADOOP-11858 . [JDK8] Set minimum version of Hadoop 3 to JDK 8. (wang: rev 4b55642b9d836691592405805c181d12c2ed7e50) hadoop-project/pom.xml pom.xml BUILDING.txt
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Andrew and Robert for the work, and thanks all who discussed in the mailing list.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Andrew and Robert for the work, and thanks all who discussed in the mailing list.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks for doing this everyone. Alonzo Church came up with the Lambda Calculus in the 1930s —it's good to be able to use it in Java.

          I think a good place to start will actually be tests; I've used closures a lot there in groovy; ScalaTest's eventually(interval, duration, closure) call does the same: lets you declare a closure which is expected to eventually succeed, time out, or raise a special "this test will never succeed" exception. It lets you spin efficiently with a fast fail and no need to hard code in long delays.

          Show
          stevel@apache.org Steve Loughran added a comment - thanks for doing this everyone. Alonzo Church came up with the Lambda Calculus in the 1930s —it's good to be able to use it in Java. I think a good place to start will actually be tests; I've used closures a lot there in groovy; ScalaTest's eventually(interval, duration, closure) call does the same: lets you declare a closure which is expected to eventually succeed, time out, or raise a special "this test will never succeed" exception. It lets you spin efficiently with a fast fail and no need to hard code in long delays.
          Hide
          andrew.wang Andrew Wang added a comment -

          I moved this to a new issue so it's called out in the release notes / changes as a new feature, hope that's cool with everyone.

          Show
          andrew.wang Andrew Wang added a comment - I moved this to a new issue so it's called out in the release notes / changes as a new feature, hope that's cool with everyone.

            People

            • Assignee:
              rkanter Robert Kanter
              Reporter:
              rkanter Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development