Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.1
    • Component/s: None
    • Labels:
      None

      Description

      As discussed on the mailinglist, Oracle removed the Sun JDK from the Ubuntu repos which caused services like Hadoop to stop working.
      Here is a patch which installs OpenJDK instead of the Sun JDK and works when using Ubuntu image, whirr.image-id=49 (Ubuntu 10.04 LTS)

      To make it work with rpms, OpenJDK needs to be uploaded to the whirr-third-party S3 bucket. Correct?

      1. WHIRR-518.patch
        1 kB
        Frank Scholten
      2. WHIRR-518.patch
        8 kB
        Frank Scholten
      3. install_java.sh
        2 kB
        Marco Didonna
      4. WHIRR-518-for-0.7.1.patch
        17 kB
        Andrei Savu
      5. WHIRR-518-for-0.7.1.patch
        17 kB
        Tom White

        Activity

        Hide
        Andrei Savu added a comment -

        Frank have you seen core/src/main/resources/functions/install_openjdk.sh ? We already have this in trunk. Can you make that the default?

        Show
        Andrei Savu added a comment - Frank have you seen core/src/main/resources/functions/install_openjdk.sh ? We already have this in trunk. Can you make that the default?
        Hide
        Frank Scholten added a comment -

        Ah yes, didn't notice that one.

        Show
        Frank Scholten added a comment - Ah yes, didn't notice that one.
        Hide
        Andrei Savu added a comment -

        +1 & committed to trunk. Tested with ZK on aws-ec2 & cloudservers-uk. Thanks Frank!

        Show
        Andrei Savu added a comment - +1 & committed to trunk. Tested with ZK on aws-ec2 & cloudservers-uk. Thanks Frank!
        Hide
        Karel Vervaeke added a comment -

        IIRC I had problems running hadoop and hbase with just whirr.java.install-function=install_openjdk.

        I also had to add
        hadoop-env.JAVA_HOME=/usr/lib/jvm/java-6-openjdk
        hbase-env.JAVA_HOME=/usr/lib/jvm/java-6-openjdk

        (the reason being that the hadoop/hbase scripts look at fixed places for detecting JAVA_HOME, but the openjdk path isn't amongst those places).

        Actually the 'export JAVA_HOME' from install_openjdk.sh should be enough, but apparently it didn't get into the environment under which hadoop/hbase is run.

        Show
        Karel Vervaeke added a comment - IIRC I had problems running hadoop and hbase with just whirr.java.install-function=install_openjdk. I also had to add hadoop-env.JAVA_HOME=/usr/lib/jvm/java-6-openjdk hbase-env.JAVA_HOME=/usr/lib/jvm/java-6-openjdk (the reason being that the hadoop/hbase scripts look at fixed places for detecting JAVA_HOME, but the openjdk path isn't amongst those places). Actually the 'export JAVA_HOME' from install_openjdk.sh should be enough, but apparently it didn't get into the environment under which hadoop/hbase is run.
        Hide
        Andrei Savu added a comment - - edited

        Karel do you think we should add them to whirr-hadoop-default.properties and whirr-hbase-default.properties? How about something like this (not tested):

        hadoop-env.JAVA_HOME=$(source /etc/profile;echo $JAVA_HOME)
        
        Show
        Andrei Savu added a comment - - edited Karel do you think we should add them to whirr-hadoop-default.properties and whirr-hbase-default.properties? How about something like this (not tested): hadoop-env.JAVA_HOME=$(source /etc/profile;echo $JAVA_HOME)
        Hide
        Karel Vervaeke added a comment -

        I'm good with that, but consider that adding it there makes it impossible to unset (unless you count setting it to empty-string).

        Show
        Karel Vervaeke added a comment - I'm good with that, but consider that adding it there makes it impossible to unset (unless you count setting it to empty-string).
        Hide
        Marco Didonna added a comment -

        If anyone cares about the sun-jdk I've modified the install_java function in order to install the jdk debs using this script https://github.com/flexiondotorg/oab-java6.
        It works on EC2

        Show
        Marco Didonna added a comment - If anyone cares about the sun-jdk I've modified the install_java function in order to install the jdk debs using this script https://github.com/flexiondotorg/oab-java6 . It works on EC2
        Hide
        Evan Pollan added a comment - - edited

        I'm trying to get the openjdk solution to work with the cdh hadoop stack, and I'm running into the JAVA_HOME issue.

        Karel, Andrei: if I use Andrei's solution and re-build the whirr-hadoop artifact, will this also ensure that JAVA_HOME is set for the cdh_hadoop services? Or, would that be managed by a different codepath? (forgive my lack of familiarity with the whirr codebase).

        ETA: Karel – was your fix to just add those lines to your cluster properties file? If so, that would be a simpler fix for me...

        Show
        Evan Pollan added a comment - - edited I'm trying to get the openjdk solution to work with the cdh hadoop stack, and I'm running into the JAVA_HOME issue. Karel, Andrei: if I use Andrei's solution and re-build the whirr-hadoop artifact, will this also ensure that JAVA_HOME is set for the cdh_hadoop services? Or, would that be managed by a different codepath? (forgive my lack of familiarity with the whirr codebase). ETA: Karel – was your fix to just add those lines to your cluster properties file? If so, that would be a simpler fix for me...
        Hide
        Andrei Savu added a comment -

        Evan I'm not sure if my solution works but yes if you rebuild whirr-hadoop all defaults also apply to CDH.

        Show
        Andrei Savu added a comment - Evan I'm not sure if my solution works but yes if you rebuild whirr-hadoop all defaults also apply to CDH.
        Hide
        Marco Didonna added a comment -

        Evan if you're in a hurry you can switch to my solution for the moment. It works with cdh

        Show
        Marco Didonna added a comment - Evan if you're in a hurry you can switch to my solution for the moment. It works with cdh
        Hide
        Karel Vervaeke added a comment -

        hbase-env.JAVA_HOME=... will work for cdh as well.

        Show
        Karel Vervaeke added a comment - hbase-env.JAVA_HOME=... will work for cdh as well.
        Hide
        Evan Pollan added a comment -

        Great – just to clarify, though: are those lines added to the cluster properties file?

        Show
        Evan Pollan added a comment - Great – just to clarify, though: are those lines added to the cluster properties file?
        Hide
        Andrei Savu added a comment -

        Marco let's create an install_oab_java as an alternative to openjdk. Can you create a JIRA and a patch?

        Show
        Andrei Savu added a comment - Marco let's create an install_oab_java as an alternative to openjdk. Can you create a JIRA and a patch?
        Hide
        Andrei Savu added a comment -

        Evan the end result should be the same if you add them to whirr-hadoop-defaults... or to your own .properties file.

        Show
        Andrei Savu added a comment - Evan the end result should be the same if you add them to whirr-hadoop-defaults... or to your own .properties file.
        Hide
        Marco Didonna added a comment -

        I'm on it Andrei ... please check IRC

        Show
        Marco Didonna added a comment - I'm on it Andrei ... please check IRC
        Hide
        Evan Pollan added a comment -

        Awesome. Looks like things are working with whirr @ rev 1290925 + the following additions to my cluster properties file:

        whirr.java.install-function=install_openjdk
        hadoop-env.JAVA_HOME=/usr/lib/jvm/java-6-openjdk
        

        Nothing like a minor emergency to get oneself familiarized with an open-source project's codebase

        Show
        Evan Pollan added a comment - Awesome. Looks like things are working with whirr @ rev 1290925 + the following additions to my cluster properties file: whirr.java.install-function=install_openjdk hadoop-env.JAVA_HOME=/usr/lib/jvm/java-6-openjdk Nothing like a minor emergency to get oneself familiarized with an open-source project's codebase
        Hide
        Karel Vervaeke added a comment -

        @Evan: A bit late, but as you've noticed already: yes, those lines should go into the cluster properties file

        Show
        Karel Vervaeke added a comment - @Evan: A bit late, but as you've noticed already: yes, those lines should go into the cluster properties file
        Hide
        Andrei Savu added a comment -

        Nothing like a minor emergency to get oneself familiarized with an open-source project's codebase

        Priceless moments Watch for 0.7.1 over the next few days.

        Show
        Andrei Savu added a comment - Nothing like a minor emergency to get oneself familiarized with an open-source project's codebase Priceless moments Watch for 0.7.1 over the next few days.
        Hide
        Andrei Savu added a comment -

        I'm reopening to fix the JAVA_HOME issue and commit to the 0.7 branch.

        Show
        Andrei Savu added a comment - I'm reopening to fix the JAVA_HOME issue and commit to the 0.7 branch.
        Hide
        Andrei Savu added a comment -

        I am going to add

        source /etc/profile
        

        to /etc/hadoop/conf/hadoop-env.sh.

        Show
        Andrei Savu added a comment - I am going to add source /etc/profile to /etc/hadoop/conf/hadoop-env.sh.
        Hide
        Andrei Savu added a comment -

        FYI I've got a clean integration tests run for Hadoop (services/hadoop) on aws-ec2. I'm now checking CDH.

        Show
        Andrei Savu added a comment - FYI I've got a clean integration tests run for Hadoop (services/hadoop) on aws-ec2. I'm now checking CDH.
        Hide
        Andrei Savu added a comment -

        This is a slightly updated version of the original patch for the 0.7 branch. Hadoop seems to be working fine on aws-ec2 but I haven't tested CDH yet. Wdyt?

        Show
        Andrei Savu added a comment - This is a slightly updated version of the original patch for the 0.7 branch. Hadoop seems to be working fine on aws-ec2 but I haven't tested CDH yet. Wdyt?
        Hide
        Antonio Piccolboni added a comment -

        CDH3 Works with build you made available to me – I suppose from trunk. Still lots of errors in console but gets the job done.

        Show
        Antonio Piccolboni added a comment - CDH3 Works with build you made available to me – I suppose from trunk. Still lots of errors in console but gets the job done.
        Hide
        Tom White added a comment -

        CDH ZK worked for me, but I'm having problems with CDH Hadoop. I'll look into it now.

        Show
        Tom White added a comment - CDH ZK worked for me, but I'm having problems with CDH Hadoop. I'll look into it now.
        Hide
        Tom White added a comment -

        Looks like a problem with 'source' not being recognized in sh (it's bash only) - here's a patch which changes it to '.'.

        Show
        Tom White added a comment - Looks like a problem with 'source' not being recognized in sh (it's bash only) - here's a patch which changes it to '.'.
        Hide
        Tom White added a comment -

        Integration tests pass for Hadoop and HBase for CDH on EC2 with this patch.

        +1

        Show
        Tom White added a comment - Integration tests pass for Hadoop and HBase for CDH on EC2 with this patch. +1
        Hide
        Andrei Savu added a comment -

        Tom do you think it's a good idea to release 0.7.1 with openjdk as default?

        Show
        Andrei Savu added a comment - Tom do you think it's a good idea to release 0.7.1 with openjdk as default?
        Hide
        Antonio Piccolboni added a comment -

        What is the alternative?

        On Wed, Feb 22, 2012 at 2:23 PM, Andrei Savu (Commented) (JIRA) <

        Show
        Antonio Piccolboni added a comment - What is the alternative? On Wed, Feb 22, 2012 at 2:23 PM, Andrei Savu (Commented) (JIRA) <
        Hide
        Andrei Savu added a comment -

        To install from the .tar.gz file - we are exploring this alternative in WHIRR-521.

        Show
        Andrei Savu added a comment - To install from the .tar.gz file - we are exploring this alternative in WHIRR-521 .
        Hide
        Andrei Savu added a comment -

        +1 CDH also works fine for me with this patch. I will go ahead and commit and rework WHIRR-521.

        Show
        Andrei Savu added a comment - +1 CDH also works fine for me with this patch. I will go ahead and commit and rework WHIRR-521 .
        Hide
        Tom White added a comment -

        Hadoop hasn't been well tested against OpenJDK, so I'm a little reluctant - although the limited testing I've done hasn't uncovered any problems. Also, folks can use the install_oab_java.sh script which will be in 0.7.1 if they want to use Sun JDK 1.6, although that has other problems (e.g. it's slower to install). Installing from a .tar.gz would be a good alternative too, so it's great to hear that's being worked on - the more options the better until a solid solution emerges.

        Show
        Tom White added a comment - Hadoop hasn't been well tested against OpenJDK, so I'm a little reluctant - although the limited testing I've done hasn't uncovered any problems. Also, folks can use the install_oab_java.sh script which will be in 0.7.1 if they want to use Sun JDK 1.6, although that has other problems (e.g. it's slower to install). Installing from a .tar.gz would be a good alternative too, so it's great to hear that's being worked on - the more options the better until a solid solution emerges.
        Hide
        Andrei Savu added a comment -

        Also committed to the 0.7 branch.

        Show
        Andrei Savu added a comment - Also committed to the 0.7 branch.
        Hide
        Marco Didonna added a comment -

        @Tom White what about jdk 7? is it safe to use it with hadoop (and its stack) ?

        Show
        Marco Didonna added a comment - @Tom White what about jdk 7? is it safe to use it with hadoop (and its stack) ?
        Hide
        Tom White added a comment -

        Marco - Hadoop hasn't been tested extensively with JDK 7 yet either, so it's good to have a Sun JDK 1.6 option until that changes.

        Show
        Tom White added a comment - Marco - Hadoop hasn't been tested extensively with JDK 7 yet either, so it's good to have a Sun JDK 1.6 option until that changes.

          People

          • Assignee:
            Frank Scholten
            Reporter:
            Frank Scholten
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development