Bigtop
  1. Bigtop
  2. BIGTOP-927

Update puppet recipes to support Ubuntu

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.7.0
    • Component/s: deployment
    • Labels:
      None

      Description

      In BIGTOP-720 we have started with a great set of recipes to automatically configure build environment. However it has al least two limitations:

      • it needs manual assistance in a form of downloading tarballs of the web
      • it doesn't support Ubuntu

      Let's fix it.

      1. 0001-BIGTOP-927.-Add-extra-ubuntu-packages-and-ppa-repo-t.patch
        7 kB
        Ian Mordey
      2. BIGTOP-927.patch
        7 kB
        Konstantin Boudnik
      3. BIGTOP-927-Add-ubuntu-support.patch
        13 kB
        Ian Mordey
      4. BIGTOP-927.patch
        11 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Committed to master as 85d1353. Thanks Ian!

          Show
          Konstantin Boudnik added a comment - Committed to master as 85d1353. Thanks Ian!
          Hide
          Konstantin Boudnik added a comment -

          Looks like I have missed deps.pp from BIGTOP-932 - I am sorry about that, Ian.

          Now, for this new patch of yours: I notices that you have removed bigtop-toolchain::deps from ant and forrest modules, which prevents them from working alone. So, it seems that removal wasn't intentional. So, I have restored these two lines and everything seems to be working under my Ubuntu 12.04.

          I am uploading new version of the patch with include restored in ant and forrest model and committing it in this form. Please let me know if I have misunderstood your intentions - we'll fix it.

          Show
          Konstantin Boudnik added a comment - Looks like I have missed deps.pp from BIGTOP-932 - I am sorry about that, Ian. Now, for this new patch of yours: I notices that you have removed bigtop-toolchain::deps from ant and forrest modules, which prevents them from working alone. So, it seems that removal wasn't intentional. So, I have restored these two lines and everything seems to be working under my Ubuntu 12.04. I am uploading new version of the patch with include restored in ant and forrest model and committing it in this form. Please let me know if I have misunderstood your intentions - we'll fix it.
          Hide
          Ian Mordey added a comment -

          Updated it so it'll patch against master and fixed up a couple of dependency issues.

          I've just ran it twice against 12.04.3 and I can make hadoop-deb.

          Show
          Ian Mordey added a comment - Updated it so it'll patch against master and fixed up a couple of dependency issues. I've just ran it twice against 12.04.3 and I can make hadoop-deb.
          Hide
          Konstantin Boudnik added a comment -

          the patch no longer applies (mainly because of the changes in bigtop-toolchain/manifests/jdk.pp). Would you mind updating it, Ian? I will commit it once it is ready. Thanks!

          Show
          Konstantin Boudnik added a comment - the patch no longer applies (mainly because of the changes in bigtop-toolchain/manifests/jdk.pp). Would you mind updating it, Ian? I will commit it once it is ready. Thanks!
          Hide
          Konstantin Boudnik added a comment - - edited

          Sure, no hurry! Thanks for helping out with this PITA. Have a good vacation too!

          Show
          Konstantin Boudnik added a comment - - edited Sure, no hurry! Thanks for helping out with this PITA. Have a good vacation too!
          Hide
          Ian Mordey added a comment -

          I may not have my dependencies properly in order. There's a couple of execs that are supposed to accept the oracle license before installing the oracle package. It looks like either one or both of these didn't run before the package attempted to install. Silly Oracle. How's OpenJDK working these days?

          Anyway. I'm off on vacation for a week so won't be able to take a look at this till I get back. I don't think you should commit as-is. I'd rather do some heavier testing and also make sure OpenSuSe/SLES work properly. If you need to release 0.6.0 and you feel this is good enough then feel free to do so and I'll work on a better way when I get back.

          Show
          Ian Mordey added a comment - I may not have my dependencies properly in order. There's a couple of execs that are supposed to accept the oracle license before installing the oracle package. It looks like either one or both of these didn't run before the package attempted to install. Silly Oracle. How's OpenJDK working these days? Anyway. I'm off on vacation for a week so won't be able to take a look at this till I get back. I don't think you should commit as-is. I'd rather do some heavier testing and also make sure OpenSuSe/SLES work properly. If you need to release 0.6.0 and you feel this is good enough then feel free to do so and I'll work on a better way when I get back.
          Hide
          Konstantin Boudnik added a comment -

          I have changed the notice for Ubuntu wrt the protobuf.
          Cleaned up some minor whitespace changes. I think the patch is ready to go.

          +1

          Show
          Konstantin Boudnik added a comment - I have changed the notice for Ubuntu wrt the protobuf. Cleaned up some minor whitespace changes. I think the patch is ready to go. +1
          Hide
          Konstantin Boudnik added a comment - - edited

          for protobuf it says "Not required on Ubuntu". I would change the message to something: "Ubuntu provides a standard package" cause it sounds confusing.

          I ran new puppet on pretty much freshly minted 12.04 and it was able to install all the packages but:

          Unpacking java-common (from .../java-common_0.43ubuntu2_all.deb) ...
          Selecting previously unselected package oracle-java6-installer.
          Unpacking oracle-java6-installer (from .../oracle-java6-installer_6u45-0~webupd8~0_all.deb) ...
          
          oracle-license-v1-1 license could not be presented
          try 'dpkg-reconfigure debconf' to select a frontend other than noninteractive
          
          dpkg: error processing /var/cache/apt/archives/oracle-java6-installer_6u45-0~webupd8~0_all.deb (--unpack):
           subprocess new pre-installation script returned error exit status 2
          No apport report written because MaxReports is reached already
          Selecting previously unselected package gsfonts-x11.
          Unpacking gsfonts-x11 (from .../gsfonts-x11_0.22_all.deb) ...
          Processing triggers for doc-base ...
          Processing 2 added doc-base files...
          Processing triggers for man-db ...
          Processing triggers for fontconfig ...
          Errors were encountered while processing:
           /var/cache/apt/archives/oracle-java6-installer_6u45-0~webupd8~0_all.deb
          E: Sub-process /usr/bin/dpkg returned an error code (1)
          

          This seemed to leave the dpkg system in a strange state, where dpkg lock has been left behind. Cleaning that and running apply for the second time:

          err: /Stage[main]/Bigtop-toolchain::Jdk/Package[oracle-java6-installer]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install oracle-java6-installer' returned 100: E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
          
          err: /Stage[main]/Bigtop-toolchain::Packages/Package[asciidoc]/ensure: change from absent to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install asciidoc' returned 100: E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
          

          Running sudo dpkg --configure -a a couple of time in the row seems to address the problems with asciidoc - what a heavyweight if you ask me.

          And another repeat of apply put everything as expected. I guess the oracle repo is a bitch to blame, because the connection there seemed to be pretty slow (may be just today).

          I am running make hadoop-deb sqoop-deb - all seems to be going fine so far.

          Show
          Konstantin Boudnik added a comment - - edited for protobuf it says "Not required on Ubuntu". I would change the message to something: "Ubuntu provides a standard package" cause it sounds confusing. I ran new puppet on pretty much freshly minted 12.04 and it was able to install all the packages but: Unpacking java-common (from .../java-common_0.43ubuntu2_all.deb) ... Selecting previously unselected package oracle-java6-installer. Unpacking oracle-java6-installer (from .../oracle-java6-installer_6u45-0~webupd8~0_all.deb) ... oracle-license-v1-1 license could not be presented try 'dpkg-reconfigure debconf' to select a frontend other than noninteractive dpkg: error processing /var/cache/apt/archives/oracle-java6-installer_6u45-0~webupd8~0_all.deb (--unpack): subprocess new pre-installation script returned error exit status 2 No apport report written because MaxReports is reached already Selecting previously unselected package gsfonts-x11. Unpacking gsfonts-x11 (from .../gsfonts-x11_0.22_all.deb) ... Processing triggers for doc-base ... Processing 2 added doc-base files... Processing triggers for man-db ... Processing triggers for fontconfig ... Errors were encountered while processing: /var/cache/apt/archives/oracle-java6-installer_6u45-0~webupd8~0_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) This seemed to leave the dpkg system in a strange state, where dpkg lock has been left behind. Cleaning that and running apply for the second time: err: /Stage[main]/Bigtop-toolchain::Jdk/Package[oracle-java6-installer]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install oracle-java6-installer' returned 100: E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. err: /Stage[main]/Bigtop-toolchain::Packages/Package[asciidoc]/ensure: change from absent to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install asciidoc' returned 100: E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. Running sudo dpkg --configure -a a couple of time in the row seems to address the problems with asciidoc - what a heavyweight if you ask me. And another repeat of apply put everything as expected. I guess the oracle repo is a bitch to blame, because the connection there seemed to be pretty slow (may be just today). I am running make hadoop-deb sqoop-deb - all seems to be going fine so far.
          Hide
          Ian Mordey added a comment -

          New patch. PPA support has been re-written and packages are now all included.

          Show
          Ian Mordey added a comment - New patch. PPA support has been re-written and packages are now all included.
          Hide
          Ian Mordey added a comment -

          I can rewrite that bit of code in a different way if it's an issue. All it does is pull in a PPA repository for Oracle Java.

          I'll add those packages, sort out the PPA and submit another patch.

          Show
          Ian Mordey added a comment - I can rewrite that bit of code in a different way if it's an issue. All it does is pull in a PPA repository for Oracle Java. I'll add those packages, sort out the PPA and submit another patch.
          Hide
          Konstantin Boudnik added a comment -

          Overall +1.

          Looks like a couple of packages are missing:

          • build-essential
          • dh-make
          • reprepro
          • automake/autoconf
          • fuse-libs
          • fuse-devel
          • lzo-devel
          • openssh-devel
          • asciidoc

          I am somewhat unsure about the license bigtop-toolchain/LICENSE. Is it compatible with ASF to add copyright clause and contribute the code to ASF? Anyone?

          Show
          Konstantin Boudnik added a comment - Overall +1. Looks like a couple of packages are missing: build-essential dh-make reprepro automake/autoconf fuse-libs fuse-devel lzo-devel openssh-devel asciidoc I am somewhat unsure about the license bigtop-toolchain/LICENSE . Is it compatible with ASF to add copyright clause and contribute the code to ASF? Anyone?
          Hide
          Ian Mordey added a comment -

          Initial attempt at ubuntu support.

          Show
          Ian Mordey added a comment - Initial attempt at ubuntu support.
          Hide
          Konstantin Boudnik added a comment -

          The list of the packages is in the top-level README file. I believe this is up to date.
          Your way of testing is pretty much sufficient, except that some of the components like Sqoop would need asciidoc, that is unique for Sqoop only.
          I think the best way would be
          make hadoop-dep sqoop-deb

          Show
          Konstantin Boudnik added a comment - The list of the packages is in the top-level README file. I believe this is up to date. Your way of testing is pretty much sufficient, except that some of the components like Sqoop would need asciidoc, that is unique for Sqoop only. I think the best way would be make hadoop-dep sqoop-deb
          Hide
          Ian Mordey added a comment -

          On the ubuntu front. I've got JDK working now. Where can I get a list of required packages for an ubuntu build and how would I go about testing? Is it just a case of install the packages and do a make hadoop-deb?

          Show
          Ian Mordey added a comment - On the ubuntu front. I've got JDK working now. Where can I get a list of required packages for an ubuntu build and how would I go about testing? Is it just a case of install the packages and do a make hadoop-deb?

            People

            • Assignee:
              Ian Mordey
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development