Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hadoop now requires Java 6.

      Description

      We should require Java 6 for release 0.17. Java 6 is now available for OS/X. Hadoop performs much better on Java 6. And, finally, there are features of Java 6 (like 'df') that would be nice to use.

      1. HADOOP-2325.patch
        8 kB
        Doug Cutting
      2. HADOOP-2325.patch
        4 kB
        Doug Cutting

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/ )
        Hide
        Doug Cutting added a comment -

        I just committed this.

        Show
        Doug Cutting added a comment - I just committed this.
        Hide
        Nigel Daley added a comment -

        Doug, the problem is that the patch testing process can't test these changes to the patch testing process itself. Hudson calls src/test/bin/test-patch.sh before it has been patched.

        I think I've got all the Hudson pieces in place for this patch. Please commit it on Monday and I'll monitor the first few builds.

        Show
        Nigel Daley added a comment - Doug, the problem is that the patch testing process can't test these changes to the patch testing process itself. Hudson calls src/test/bin/test-patch.sh before it has been patched. I think I've got all the Hudson pieces in place for this patch. Please commit it on Monday and I'll monitor the first few builds.
        Hide
        Doug Cutting added a comment -

        The problem was that Java 5 was not found, which caused some targets to fail. Nigel, can you re-check the value of -Djava5.home? Thanks!

        Show
        Doug Cutting added a comment - The problem was that Java 5 was not found, which caused some targets to fail. Nigel, can you re-check the value of -Djava5.home? Thanks!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12384854/HADOOP-2325.patch
        against trunk revision 673517.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12384854/HADOOP-2325.patch against trunk revision 673517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2792/console This message is automatically generated.
        Hide
        Nigel Daley added a comment -

        Updated Hudson. Trying patch.

        Show
        Nigel Daley added a comment - Updated Hudson. Trying patch.
        Hide
        Doug Cutting added a comment -

        Here's a new version of the patch, updated for changes to trunk. I also fixed things so that forrest (which requires Java5) will now work only if -Djava5.home is specified, and updated the test-patch.sh script accordingly.

        Nigel, if you can please switch Hudson to:

        • use Java6 for Hadoop core test runs; and
        • specify -Djava5.home=<location of JDK 1.5>

        Then I will submit this patch and I think it will pass Hudson's tests and we can commit it without breaking Hudson.

        Show
        Doug Cutting added a comment - Here's a new version of the patch, updated for changes to trunk. I also fixed things so that forrest (which requires Java5) will now work only if -Djava5.home is specified, and updated the test-patch.sh script accordingly. Nigel, if you can please switch Hudson to: use Java6 for Hadoop core test runs; and specify -Djava5.home=<location of JDK 1.5> Then I will submit this patch and I think it will pass Hudson's tests and we can commit it without breaking Hudson.
        Hide
        Owen O'Malley added a comment -

        +1 to moving forward to java 6.

        Show
        Owen O'Malley added a comment - +1 to moving forward to java 6.
        Hide
        Nigel Daley added a comment -

        Re: hudson

        • I can just configure the build to use JDK 1.6 from a picklist that Hudson is already configured with
        • I'll leave the Hadoop-branch build on JDK 1.5 while the branch is 0.18

        • I agree the pending switch should be documented in the 0.18 release notes.
        Show
        Nigel Daley added a comment - Re: hudson I can just configure the build to use JDK 1.6 from a picklist that Hudson is already configured with I'll leave the Hadoop-branch build on JDK 1.5 while the branch is 0.18 I agree the pending switch should be documented in the 0.18 release notes.
        Hide
        steve_l added a comment -

        +1 to the move, with some caveats

        • hudson has to move up first, to make sure it is happy
        • backporting becomes tricky, and that includes to 18.x.

        Stuff built with Java6 doesn't necessarily load in Java5; I've encountered problems with libraries I built myself not loading in Java5. So keeping some stuff java5 and java6 isn't going to work reliably. Best to make the jump and spell it out in advance "18.x is the last java5 version".

        Show
        steve_l added a comment - +1 to the move, with some caveats hudson has to move up first, to make sure it is happy backporting becomes tricky, and that includes to 18.x. Stuff built with Java6 doesn't necessarily load in Java5; I've encountered problems with libraries I built myself not loading in Java5. So keeping some stuff java5 and java6 isn't going to work reliably. Best to make the jump and spell it out in advance "18.x is the last java5 version".
        Hide
        Doug Cutting added a comment -

        > Would it be possible to move all the servers to java6, but keep the DFSClient on java 5?

        Maintaining that would be more painful than the benefits it would offer. We should wholly switch to Java 6 or wholly stay in Java 5.

        Show
        Doug Cutting added a comment - > Would it be possible to move all the servers to java6, but keep the DFSClient on java 5? Maintaining that would be more painful than the benefits it would offer. We should wholly switch to Java 6 or wholly stay in Java 5.
        Hide
        Michael Bieniosek added a comment -

        Would it be possible to move all the servers to java6, but keep the DFSClient on java 5?

        Show
        Michael Bieniosek added a comment - Would it be possible to move all the servers to java6, but keep the DFSClient on java 5?
        Hide
        Doug Cutting added a comment -

        Should we commit this now, early in the 0.19 cycle, so that we can address things like HADOOP-3642?

        Show
        Doug Cutting added a comment - Should we commit this now, early in the 0.19 cycle, so that we can address things like HADOOP-3642 ?
        Hide
        Nigel Daley added a comment -

        Moving to 0.19 release to continue discussion in the context of that release.

        Show
        Nigel Daley added a comment - Moving to 0.19 release to continue discussion in the context of that release.
        Hide
        Doug Cutting added a comment -

        > Doug, should we mark this for 0.19?

        Sure. Folks won't like it then either, but we should probably revisit it periodically.

        Show
        Doug Cutting added a comment - > Doug, should we mark this for 0.19? Sure. Folks won't like it then either, but we should probably revisit it periodically.
        Hide
        Nigel Daley added a comment -

        Doug, should we mark this for 0.19?

        Show
        Nigel Daley added a comment - Doug, should we mark this for 0.19?
        Hide
        Brice Arnould added a comment -

        Have you tried with IcedTea ? It's the JDK I use to build and run Hadoop on my Ubuntu/x86 system. It seems that it have been ported to MacOS 10.4 [1], and that it can be built on PowerPC [2], but I'm unsure about the combination of the two.
        [1] http://landonf.bikemonkey.org/static/soylatte/
        [2] http://langel.wordpress.com/2008/01/03/icedtea-15-released-with-ppc-support/

        Show
        Brice Arnould added a comment - Have you tried with IcedTea ? It's the JDK I use to build and run Hadoop on my Ubuntu/x86 system. It seems that it have been ported to MacOS 10.4 [1] , and that it can be built on PowerPC [2] , but I'm unsure about the combination of the two. [1] http://landonf.bikemonkey.org/static/soylatte/ [2] http://langel.wordpress.com/2008/01/03/icedtea-15-released-with-ppc-support/
        Hide
        Ted Dunning added a comment -

        Java 6 appears to not be available on OSX 10.4, either.

        I would be willing to use this as an excuse to upgrade my dev machine to 10.5, though.

        Show
        Ted Dunning added a comment - Java 6 appears to not be available on OSX 10.4, either. I would be willing to use this as an excuse to upgrade my dev machine to 10.5, though.
        Hide
        Doug Cutting added a comment -

        There is no urgency to this, but it would be nice not to delay it.

        Hadoop performs better on Java 6, so we want to encourage folks to use Java 6. The vast majority of Hadoop contributors use Java 6 when developing, and hence do run into (usually minor) back-compatibility problems, like the one we've just seen with HADOOP-3468. These problems cost the project some productivity.

        On the other hand, we don't want to leave contributors behind. It appears that Java 6 is still not available on PowerPC-based Macs. Sigh. Michael, is the G4 your primary development platform?

        Show
        Doug Cutting added a comment - There is no urgency to this, but it would be nice not to delay it. Hadoop performs better on Java 6, so we want to encourage folks to use Java 6. The vast majority of Hadoop contributors use Java 6 when developing, and hence do run into (usually minor) back-compatibility problems, like the one we've just seen with HADOOP-3468 . These problems cost the project some productivity. On the other hand, we don't want to leave contributors behind. It appears that Java 6 is still not available on PowerPC-based Macs. Sigh. Michael, is the G4 your primary development platform?
        Hide
        steve_l added a comment -

        It should be fairly trivial to run File.getFreeSpace() using introspection; what is equally interesting is the java6 script engine and built in javascript interpreter; you could add support for mappers and reducers written in JS or another supported language.

        I would strongly argue against anyone using JAXB or the JAX-WS runtime built into Java6, as the exact version varies from point release of the JVM.. java 6 U4 and later provides a different JAXB runtime, just to break everyone's code. This effectively means 'dont use JAXB or JAX-WS'.

        Show
        steve_l added a comment - It should be fairly trivial to run File.getFreeSpace() using introspection; what is equally interesting is the java6 script engine and built in javascript interpreter; you could add support for mappers and reducers written in JS or another supported language. I would strongly argue against anyone using JAXB or the JAX-WS runtime built into Java6, as the exact version varies from point release of the JVM.. java 6 U4 and later provides a different JAXB runtime, just to break everyone's code. This effectively means 'dont use JAXB or JAX-WS'.
        Hide
        Ankur added a comment -

        The patch was provided by me and it had the necessary jars (Commons net & Apache Mina FTP server) created after compiling the relevant classes under Java 1.6 on my local linux machine. While creation of commons net jars on local machine was unnecessary, I had to create Mina FTP server jars locally as it does not have an official release yet.

        I have fixed problem by picking up commons net jar from the download site and re-compiling Mina FTP server classes under Java 1.5. The jars have been packaged and attached on both HADOOP-3468 and HADOOP-3246. There should be no compilation errors now.

        My sincere apologies for the inconvenience caused to everybody.

        Show
        Ankur added a comment - The patch was provided by me and it had the necessary jars (Commons net & Apache Mina FTP server) created after compiling the relevant classes under Java 1.6 on my local linux machine. While creation of commons net jars on local machine was unnecessary, I had to create Mina FTP server jars locally as it does not have an official release yet. I have fixed problem by picking up commons net jar from the download site and re-compiling Mina FTP server classes under Java 1.5. The jars have been packaged and attached on both HADOOP-3468 and HADOOP-3246 . There should be no compilation errors now. My sincere apologies for the inconvenience caused to everybody.
        Hide
        Nigel Daley added a comment -

        Looks like HADOOP-3246 (which was committed today) introduced a dependency on Java 1.6. Doug, as committer, was that intended? Hudson builds will fail until this is resolved. FWIW, my current Mac build machine is a G4 and thus can't run Java 1.6.

        Show
        Nigel Daley added a comment - Looks like HADOOP-3246 (which was committed today) introduced a dependency on Java 1.6. Doug, as committer, was that intended? Hudson builds will fail until this is resolved. FWIW, my current Mac build machine is a G4 and thus can't run Java 1.6.
        Hide
        Michael Bieniosek added a comment -

        -1

        I don't see a need for this.

        Show
        Michael Bieniosek added a comment - -1 I don't see a need for this.
        Hide
        Owen O'Malley added a comment -

        +1

        Show
        Owen O'Malley added a comment - +1
        Hide
        Michael Bieniosek added a comment -

        I'm not sure I understand the point of this patch: are there any actual dependencies on java6 we want to introduce? Users can certainly run hadoop in java6 vms if they wish, but making it impossible for mac 10.4 desktop users to use hadoop is a big problem for me.

        Show
        Michael Bieniosek added a comment - I'm not sure I understand the point of this patch: are there any actual dependencies on java6 we want to introduce? Users can certainly run hadoop in java6 vms if they wish, but making it impossible for mac 10.4 desktop users to use hadoop is a big problem for me.
        Hide
        Tsz Wo Nicholas Sze added a comment -
        Show
        Tsz Wo Nicholas Sze added a comment - Hudson indeed uses Java 1.5 since it cannot compile hadoop now. See http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2522/artifact/trunk/current/trunkJavacWarnings.txt
        Hide
        Doug Cutting added a comment -

        This may fail, if Hudson uses Java 1.5.

        Show
        Doug Cutting added a comment - This may fail, if Hudson uses Java 1.5.
        Hide
        Doug Cutting added a comment -

        This patch updates Hadoop to require Java 1.6.

        Show
        Doug Cutting added a comment - This patch updates Hadoop to require Java 1.6.
        Hide
        Michael Bieniosek added a comment -

        java 6 is available on 10.5, but not 10.4 (which I still use).... Is there any particular reason we want to force java 6? File.getFreeSpace doesn't seem like a good reason if there is such a simple workaround.

        Show
        Michael Bieniosek added a comment - java 6 is available on 10.5, but not 10.4 (which I still use).... Is there any particular reason we want to force java 6? File.getFreeSpace doesn't seem like a good reason if there is such a simple workaround.
        Hide
        Doug Cutting added a comment -

        > Should we force Java 6 in 0.17?

        I think it's too late for 0.17, since that's already been branched, and we should no longer make incompatible changes.

        Show
        Doug Cutting added a comment - > Should we force Java 6 in 0.17? I think it's too late for 0.17, since that's already been branched, and we should no longer make incompatible changes.
        Hide
        Enis Soztutar added a comment -

        Not an apple fanboy, but I think java6 is officially released : http://docs.info.apple.com/article.html?artnum=307403
        I'm strongly +1 for java 6 in 0.17.
        Should we force Java 6 in 0.17? If so we should mark this blocker !

        Show
        Enis Soztutar added a comment - Not an apple fanboy, but I think java6 is officially released : http://docs.info.apple.com/article.html?artnum=307403 I'm strongly +1 for java 6 in 0.17. Should we force Java 6 in 0.17? If so we should mark this blocker !
        Hide
        Michael Bieniosek added a comment -

        > Do we believe that the licensing situation is likely to change in the near future?

        I think we just have to wait for Apple to do an "official" 1.6 release for OSX. There was a lot of speculation that they would release shortly after OSX10.5 came out, but then they didn't.

        Show
        Michael Bieniosek added a comment - > Do we believe that the licensing situation is likely to change in the near future? I think we just have to wait for Apple to do an "official" 1.6 release for OSX. There was a lot of speculation that they would release shortly after OSX10.5 came out, but then they didn't.
        Hide
        Nigel Daley added a comment -

        This didn't pan out for 0.16.0. Assigning to 0.17.0 for another crack at it.

        Show
        Nigel Daley added a comment - This didn't pan out for 0.16.0. Assigning to 0.17.0 for another crack at it.
        Hide
        Doug Cutting added a comment -

        I certainly agree that we should strongly encourage Java 6, but I'm not yet sure that we must require it. The primary Java6 feature that's been identified as useful is File.getFreeSpace(), but this will not totally replace df, since Java still doesn't give the mount point, and we'll still need to invoke lots of other shell commands. So, assuming HADOOP-2344 fixes the current df problems, is there a reason to require Java 6?

        Show
        Doug Cutting added a comment - I certainly agree that we should strongly encourage Java 6, but I'm not yet sure that we must require it. The primary Java6 feature that's been identified as useful is File.getFreeSpace(), but this will not totally replace df, since Java still doesn't give the mount point, and we'll still need to invoke lots of other shell commands. So, assuming HADOOP-2344 fixes the current df problems, is there a reason to require Java 6?
        Hide
        Sameer Paranjpye added a comment -


        Do we believe that the licensing situation is likely to change in the near future? If not, we should perhaps do what Michael suggests.

        Show
        Sameer Paranjpye added a comment - Do we believe that the licensing situation is likely to change in the near future? If not, we should perhaps do what Michael suggests.
        Hide
        Doug Cutting added a comment -

        If Java 6 is not yet available to our Mac-based developers, then we should probably delay implementing this to a subsequent release.

        Show
        Doug Cutting added a comment - If Java 6 is not yet available to our Mac-based developers, then we should probably delay implementing this to a subsequent release.
        Hide
        Michael Bieniosek added a comment -

        Because java 6 (compiler + runtime) is not available under acceptable license on all platforms, I think references to functions like File.getFreeSpace should be done in a way that is backwards compatible with java 5. I suggest you put all the references to java 6 functions in an optional jar, with fallback methods for java 5 users.

        Show
        Michael Bieniosek added a comment - Because java 6 (compiler + runtime) is not available under acceptable license on all platforms, I think references to functions like File.getFreeSpace should be done in a way that is backwards compatible with java 5. I suggest you put all the references to java 6 functions in an optional jar, with fallback methods for java 5 users.
        Hide
        Amar Kamat added a comment -

        Using File.getFreeSpace() and File.getUsableSpace() instead of 'df -k' is a part of this issue. Refer HADOOP-2231 for further details.

        Show
        Amar Kamat added a comment - Using File.getFreeSpace() and File.getUsableSpace() instead of 'df -k' is a part of this issue. Refer HADOOP-2231 for further details.
        Hide
        Michael Bieniosek added a comment - - edited

        To quote from the bikemonkey link:

        By downloading these binaries, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp, and submit your license acceptance to Sun.

        Licensing

        The Mac OS X work is based heavily on the BSD Java port, which is licensed under the JRL. The BSDs develop Java under the JRL; FreeBSD has negotiated a license with Sun to distribute FreeBSD Java binaries based on the JRL sources.

        As the Mac port stabilizes, I am merging my work upstream into the BSD port, and in turn, it is a goal of the FreeBSD Java project to merge their work into OpenJDK. I've signed a Sun Contributor Agreement in preparation for this, and an OpenJDK Porters group has been proposed: http://thread.gmane.org/gmane.comp.java.openjdk.general/630

        While the JRL makes this initial port possible, OpenJDK's GPLv2+CE licensing makes development and distribution far simpler. I hope to contribute this work to OpenJDK as soon as is feasible.

        So, while IANAL, it appears this can only be used under JRL. But JRL only permits Research Use, defining it as:

        "Research Use" means research, evaluation, or development for the
        purpose of advancing knowledge, teaching, learning, or customizing the
        Technology or Modifications for personal use. Research Use expressly
        excludes use or distribution for direct or indirect commercial
        (including strategic) gain or advantage.

        So I suspect that running hadoop client libraries at a commercial search engine does not fall under Research Use.

        Show
        Michael Bieniosek added a comment - - edited To quote from the bikemonkey link: By downloading these binaries, you certify that you are a Licensee in good standing under the Java Research License of the Java 2 SDK, and that your access, use, and distribution of code and information you may obtain at this site is subject to the License. Please review the license at http://java.net/jrl.csp , and submit your license acceptance to Sun. Licensing The Mac OS X work is based heavily on the BSD Java port, which is licensed under the JRL. The BSDs develop Java under the JRL; FreeBSD has negotiated a license with Sun to distribute FreeBSD Java binaries based on the JRL sources. As the Mac port stabilizes, I am merging my work upstream into the BSD port, and in turn, it is a goal of the FreeBSD Java project to merge their work into OpenJDK. I've signed a Sun Contributor Agreement in preparation for this, and an OpenJDK Porters group has been proposed: http://thread.gmane.org/gmane.comp.java.openjdk.general/630 While the JRL makes this initial port possible, OpenJDK's GPLv2+CE licensing makes development and distribution far simpler. I hope to contribute this work to OpenJDK as soon as is feasible. So, while IANAL, it appears this can only be used under JRL. But JRL only permits Research Use, defining it as: "Research Use" means research, evaluation, or development for the purpose of advancing knowledge, teaching, learning, or customizing the Technology or Modifications for personal use. Research Use expressly excludes use or distribution for direct or indirect commercial (including strategic) gain or advantage. So I suspect that running hadoop client libraries at a commercial search engine does not fall under Research Use.
        Hide
        Michael Bieniosek added a comment -

        That java6 requires agreement to the java research license, which is unacceptable for commercial use.

        http://java.net/jrl.csp

        Show
        Michael Bieniosek added a comment - That java6 requires agreement to the java research license, which is unacceptable for commercial use. http://java.net/jrl.csp
        Hide
        Nigel Daley added a comment -

        Has anyone tried out the java 6 from http://landonf.bikemonkey.org/code/macosx/ ?

        To be a little more direct....

        Lack of Java 6 support on MacOS has been cited in the past as a reason to hold back on requiring Java 6 for Hadoop. If you develop on the Mac, you should try out Owen's suggested open source Java 6 implementation with your IDE and post your feedback here.

        Show
        Nigel Daley added a comment - Has anyone tried out the java 6 from http://landonf.bikemonkey.org/code/macosx/ ? To be a little more direct.... Lack of Java 6 support on MacOS has been cited in the past as a reason to hold back on requiring Java 6 for Hadoop. If you develop on the Mac, you should try out Owen's suggested open source Java 6 implementation with your IDE and post your feedback here.
        Hide
        Owen O'Malley added a comment -

        Has anyone tried out the java 6 from http://landonf.bikemonkey.org/code/macosx/ ?

        Show
        Owen O'Malley added a comment - Has anyone tried out the java 6 from http://landonf.bikemonkey.org/code/macosx/ ?

          People

          • Assignee:
            Doug Cutting
            Reporter:
            Doug Cutting
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development