Bigtop
  1. Bigtop
  2. BIGTOP-1063

Add Scala 2.9.3 to bigtop-toolchain

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: general
    • Labels:
      None

      Description

      In order to build Spark and later Shark we need to have Scala 2.9.3 in the development environment. Let's add it to new shiny bigtop-toolchain.

      1. BIGTOP-1063.patch
        50 kB
        Konstantin Boudnik
      2. BIGTOP-1063.patch
        50 kB
        Konstantin Boudnik
      3. BIGTOP-1063.patch
        50 kB
        Konstantin Boudnik
      4. BIGTOP-1063.patch
        53 kB
        Ian Mordey

        Activity

        Hide
        Roman Shaposhnik added a comment -

        This would be extremely helpful to have around for our upcoming Bigtop 0.7.0 release. Ian, any estimate on when this can be done?

        Show
        Roman Shaposhnik added a comment - This would be extremely helpful to have around for our upcoming Bigtop 0.7.0 release. Ian, any estimate on when this can be done?
        Hide
        Ian Mordey added a comment -

        I should be able to get this done this week if the latest BIGTOP-927 patch is accepted. Konstantin Boudnik Will you have time to test it soon?

        Show
        Ian Mordey added a comment - I should be able to get this done this week if the latest BIGTOP-927 patch is accepted. Konstantin Boudnik Will you have time to test it soon?
        Hide
        Konstantin Boudnik added a comment -

        reviewed, updated and commited.

        Show
        Konstantin Boudnik added a comment - reviewed, updated and commited.
        Hide
        Ian Mordey added a comment -

        This has required quite a bit of refactoring..

        I've had to rename to bigtop_toolchain due to this puppet bug: http://projects.puppetlabs.com/issues/5268

        This bug stops me from carrying variables across classes because the class name includes a hyphen!

        I've fixed a few issues the ubuntu support introduced in Centos and this now seems to work cleanly on both Ubuntu 12.04 and Centos 6.4.

        Not sure how to do this without a rename of the module. Let me know if this isn't suitable and I'll try and work around it.

        Show
        Ian Mordey added a comment - This has required quite a bit of refactoring.. I've had to rename to bigtop_toolchain due to this puppet bug: http://projects.puppetlabs.com/issues/5268 This bug stops me from carrying variables across classes because the class name includes a hyphen! I've fixed a few issues the ubuntu support introduced in Centos and this now seems to work cleanly on both Ubuntu 12.04 and Centos 6.4. Not sure how to do this without a rename of the module. Let me know if this isn't suitable and I'll try and work around it.
        Hide
        Konstantin Boudnik added a comment -

        Renaming is fine - I don't see any problem with that as far as it helps with the crazy bug.

        I will try the rest of the new toolchain later today or so and commit it if everything's good. Thanks for working on this, Ian!

        Show
        Konstantin Boudnik added a comment - Renaming is fine - I don't see any problem with that as far as it helps with the crazy bug. I will try the rest of the new toolchain later today or so and commit it if everything's good. Thanks for working on this, Ian!
        Hide
        Konstantin Boudnik added a comment -

        I have update README file to reflect new module name.

        Also, on Ubuntu, when I am running
        puppet apply -d --noop --modulepath=`pwd` -e "include bigtop_toolchain::scala"

        I am getting the following error message:

        Could not find dependency Package[oracle-java6-installer] for Exec[/usr/bin/dpkg -i /usr/src/scala-2.9.3.deb] at /home/cos/workspaces/bigtop/bigtop_toolchain/manifests/scala.pp:34

        Show
        Konstantin Boudnik added a comment - I have update README file to reflect new module name. Also, on Ubuntu, when I am running puppet apply -d --noop --modulepath=`pwd` -e "include bigtop_toolchain::scala" I am getting the following error message: Could not find dependency Package [oracle-java6-installer] for Exec [/usr/bin/dpkg -i /usr/src/scala-2.9.3.deb] at /home/cos/workspaces/bigtop/bigtop_toolchain/manifests/scala.pp:34
        Hide
        Ian Mordey added a comment -

        This is because the JDK package isn't being included. I think we should either include bigtop_toolchain::jdk in deps (as it's a dependency really) or include it in the scala class. Either way should fix the issue.

        Cheers

        Show
        Ian Mordey added a comment - This is because the JDK package isn't being included. I think we should either include bigtop_toolchain::jdk in deps (as it's a dependency really) or include it in the scala class. Either way should fix the issue. Cheers
        Hide
        Konstantin Boudnik added a comment -

        Agree. Including jdk into deps makes total sense and it fixes the situation. I've uploaded fixed version of patch. I am gonna do some more testing on CentOS and will commit it if everything is ok.

        Show
        Konstantin Boudnik added a comment - Agree. Including jdk into deps makes total sense and it fixes the situation. I've uploaded fixed version of patch. I am gonna do some more testing on CentOS and will commit it if everything is ok.
        Hide
        Konstantin Boudnik added a comment -

        Looks like on centos though, the recipes still rely on having jdk package at central puppet repository.

        Show
        Konstantin Boudnik added a comment - Looks like on centos though, the recipes still rely on having jdk package at central puppet repository.
        Hide
        Ian Mordey added a comment -

        Oracle won't let me redistribute the JDK. Is OpenJDK ready yet? That would make things a lot easier..

        You can hack around it by settings some cookies in Curl (that's what the Ubuntu PPA does) but I'm not sure if that'll get you in trouble with Oracle as you've not accepted the license. IANAL but it may well be we're already breaking the license by not forcing the user to manually accept the license on Ubuntu..

        Could we add something in the README that says by running this you accept the Oracle JDK license? If this is acceptable/legal we could then use the Curl cookie trick to download the JDK rpm files.

        Show
        Ian Mordey added a comment - Oracle won't let me redistribute the JDK. Is OpenJDK ready yet? That would make things a lot easier.. You can hack around it by settings some cookies in Curl (that's what the Ubuntu PPA does) but I'm not sure if that'll get you in trouble with Oracle as you've not accepted the license. IANAL but it may well be we're already breaking the license by not forcing the user to manually accept the license on Ubuntu.. Could we add something in the README that says by running this you accept the Oracle JDK license? If this is acceptable/legal we could then use the Curl cookie trick to download the JDK rpm files.
        Hide
        Konstantin Boudnik added a comment -

        So, you're saying that we just need to have Oracle JDK rpm file under files/ somewhere and be done with it? That's fine, I think. Let's document this requirement and I will commit the patch, one it is updated.

        On the other hand, there are tricks like
        http://stackoverflow.com/questions/10268583/how-to-automate-download-and-instalation-of-java-jdk-on-linux

        but I can't make them work for JDK6, because it is delivered through archives now.

        Another alternative is to use Open JDK for the development, but that is a totally different story.

        Show
        Konstantin Boudnik added a comment - So, you're saying that we just need to have Oracle JDK rpm file under files/ somewhere and be done with it? That's fine, I think. Let's document this requirement and I will commit the patch, one it is updated. On the other hand, there are tricks like http://stackoverflow.com/questions/10268583/how-to-automate-download-and-instalation-of-java-jdk-on-linux but I can't make them work for JDK6, because it is delivered through archives now. Another alternative is to use Open JDK for the development, but that is a totally different story.
        Hide
        Ian Mordey added a comment -

        Yes. Put the JDK rpm in files/ and it should all work.

        That should already be in the README file somewhere. It's been a requirement from the start due to redistribution issues.

        Show
        Ian Mordey added a comment - Yes. Put the JDK rpm in files/ and it should all work. That should already be in the README file somewhere. It's been a requirement from the start due to redistribution issues.
        Hide
        Konstantin Boudnik added a comment -

        yes, you're right - just spotted it. I will update the requirements section of the README as it refers to .u43 of JDK; validate and commit it afterwards.

        Show
        Konstantin Boudnik added a comment - yes, you're right - just spotted it. I will update the requirements section of the README as it refers to .u43 of JDK; validate and commit it afterwards.
        Hide
        Konstantin Boudnik added a comment -

        I have checked that placing expected RPM under files satisfy all the dependencies. I have update the patch with README.md file fixes also.

        Show
        Konstantin Boudnik added a comment - I have checked that placing expected RPM under files satisfy all the dependencies. I have update the patch with README.md file fixes also.
        Hide
        Konstantin Boudnik added a comment -

        Pushed to master as 4f87587. Thanks Ian!

        Show
        Konstantin Boudnik added a comment - Pushed to master as 4f87587. Thanks Ian!

          People

          • Assignee:
            Ian Mordey
            Reporter:
            Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development