Bigtop
  1. Bigtop
  2. BIGTOP-927 Update puppet recipes to support Ubuntu
  3. BIGTOP-932

Allow puppet recipes to automatically pull in needed files from web

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.7.0
    • Component/s: deployment

      Description

      I am splitting this from the parent task to avoid confusion.
      The idea is to be able to automatically pull in files from the internet that are needed for bigtop-toolchain.

      1. 0001-BIGTOP-932-Autodownload-packages-bump-JDK-and-ant-ve.patch
        9 kB
        Ian Mordey
      2. BIGTOP-932.patch
        6 kB
        Konstantin Boudnik
      3. BIGTOP-932.patch
        1 kB
        Konstantin Boudnik
      4. BIGTOP-932-new.patch
        8 kB
        Konstantin Boudnik
      5. BIGTOP-932-new.patch
        8 kB
        Konstantin Boudnik
      6. BIGTOP-932-new.patch
        7 kB
        Ian Mordey

        Activity

        Konstantin Boudnik created issue -
        Hide
        Konstantin Boudnik added a comment -

        Here's an indication of the direction I am thinking of going with to provide for automatic download of files needed by bigtop-toolchain.

        The issue I see with it is if someone has the server where these files are cached then, this approach won't work, as I can not use source in the file definition (see patch).

        Is there a way to be able to get away with a middle-ground approach where if a file is pre-downloaded then use it, or - if not present - download it as proposed?

        Show
        Konstantin Boudnik added a comment - Here's an indication of the direction I am thinking of going with to provide for automatic download of files needed by bigtop-toolchain. The issue I see with it is if someone has the server where these files are cached then, this approach won't work, as I can not use source in the file definition (see patch). Is there a way to be able to get away with a middle-ground approach where if a file is pre-downloaded then use it, or - if not present - download it as proposed?
        Konstantin Boudnik made changes -
        Field Original Value New Value
        Attachment BIGTOP-932.patch [ 12579261 ]
        Konstantin Boudnik made changes -
        Assignee Konstantin Boudnik [ cos ]
        Hide
        Konstantin Boudnik added a comment -

        Essentially the same as earlier, but supports ant, mvn, forrest now. Verified to to work on Ubuntu.

        The question about bandwidth preservation and the source is still open.

        Show
        Konstantin Boudnik added a comment - Essentially the same as earlier, but supports ant, mvn, forrest now. Verified to to work on Ubuntu. The question about bandwidth preservation and the source is still open.
        Konstantin Boudnik made changes -
        Attachment BIGTOP-932.patch [ 12579623 ]
        Roman Shaposhnik made changes -
        Fix Version/s backlog [ 12324373 ]
        Fix Version/s 0.6.0 [ 12323895 ]
        Konstantin Boudnik made changes -
        Labels hackathon-7-2013
        Konstantin Boudnik made changes -
        Fix Version/s 0.7.0 [ 12324362 ]
        Fix Version/s backlog [ 12324373 ]
        Hide
        Peter Linnell added a comment -

        Looking at the patch do we want to be using forrest 0.9 ? Seemed to be badly packaged when I tried it and reverted to 0.8.0. Thoughts ?

        Show
        Peter Linnell added a comment - Looking at the patch do we want to be using forrest 0.9 ? Seemed to be badly packaged when I tried it and reverted to 0.8.0. Thoughts ?
        Hide
        Konstantin Boudnik added a comment -

        Assigning this to Ian as he has a working solution.

        Show
        Konstantin Boudnik added a comment - Assigning this to Ian as he has a working solution.
        Konstantin Boudnik made changes -
        Assignee Konstantin Boudnik [ cos ] Ian Mordey [ imordey ]
        Hide
        Ian Mordey added a comment -

        Patch to auto download packages. Also bumps the JDK and Ant versions to latest.

        Show
        Ian Mordey added a comment - Patch to auto download packages. Also bumps the JDK and Ant versions to latest.
        Ian Mordey made changes -
        Hide
        Konstantin Boudnik added a comment -

        The patch looks sane and much simpler than mine initial stab at it. Hence
        +1

        I will try to run it on a fresh VM tomorrow and if it works as I expect it to, I will commit it.

        Show
        Konstantin Boudnik added a comment - The patch looks sane and much simpler than mine initial stab at it. Hence +1 I will try to run it on a fresh VM tomorrow and if it works as I expect it to, I will commit it.
        Hide
        Ian Mordey added a comment -

        One thing I would say is maybe update the README to say if you already have the ant/maven/forrest source tars and you put them in /usr/src it won't download them again. Does that satisfy your question about bandwidth preservation?

        Show
        Ian Mordey added a comment - One thing I would say is maybe update the README to say if you already have the ant/maven/forrest source tars and you put them in /usr/src it won't download them again. Does that satisfy your question about bandwidth preservation?
        Hide
        Konstantin Boudnik added a comment -

        Does the latest patch requires a particular version of the Puppet or something?
        I am getting
        Error: Could not find dependency Exec[/usr/bin/wget ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz] for Exec[/bin/tar xvzf /usr/src/apache-maven-3.0.5-bin.tar.gz

        Show
        Konstantin Boudnik added a comment - Does the latest patch requires a particular version of the Puppet or something? I am getting Error: Could not find dependency Exec [/usr/bin/wget ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz] for Exec[/bin/tar xvzf /usr/src/apache-maven-3.0.5-bin.tar.gz
        Hide
        Ian Mordey added a comment -

        They shouldn't do. What command did you run? If you ran just 'include bigtop-toolchain::maven' it probably won't pick up the deps module. We can do it two ways:
        1. Include the deps download within the individual module
        2. Include the whole deps class within each individual module.

        Option 1 seems more sensible to me. I've updated the patch to do this.

        Show
        Ian Mordey added a comment - They shouldn't do. What command did you run? If you ran just 'include bigtop-toolchain::maven' it probably won't pick up the deps module. We can do it two ways: 1. Include the deps download within the individual module 2. Include the whole deps class within each individual module. Option 1 seems more sensible to me. I've updated the patch to do this.
        Ian Mordey made changes -
        Attachment BIGTOP-932-new.patch [ 12598613 ]
        Ian Mordey made changes -
        Attachment BIGTOP-932-new.patch [ 12598613 ]
        Hide
        Ian Mordey added a comment -

        Try again. The last one had an error in the installer.pp

        Show
        Ian Mordey added a comment - Try again. The last one had an error in the installer.pp
        Ian Mordey made changes -
        Attachment BIGTOP-932-new.patch [ 12598614 ]
        Hide
        Konstantin Boudnik added a comment -

        Seems like you've copied downloads to the deps module, but haven't removed them from the others not included deps into the modules (if this was your intention).

        Show
        Konstantin Boudnik added a comment - Seems like you've copied downloads to the deps module, but haven't removed them from the others not included deps into the modules (if this was your intention).
        Hide
        Konstantin Boudnik added a comment - - edited

        Also, it seems that

        package {"wget":
          ensure => latest,
        }
        

        would be in order, otherwise the recipes choke, unless installer module is used always. Hence I am including packages to deps - I know... that's ugly ;(

        Show
        Konstantin Boudnik added a comment - - edited Also, it seems that package {"wget": ensure => latest, } would be in order, otherwise the recipes choke, unless installer module is used always. Hence I am including packages to deps - I know... that's ugly ;(
        Hide
        Konstantin Boudnik added a comment -

        Fixing minor issues with requirements and inclusion of the deps.

        +1 otherwise. I will commit if there's no objections.

        Show
        Konstantin Boudnik added a comment - Fixing minor issues with requirements and inclusion of the deps. +1 otherwise. I will commit if there's no objections.
        Konstantin Boudnik made changes -
        Attachment BIGTOP-932-new.patch [ 12598683 ]
        Konstantin Boudnik made changes -
        Attachment BIGTOP-932-new.patch [ 12598684 ]
        Hide
        Konstantin Boudnik added a comment -

        Just committed it to the master. Thanks Ian.

        Show
        Konstantin Boudnik added a comment - Just committed it to the master. Thanks Ian.
        Konstantin Boudnik made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        130d 18h 7m 1 Konstantin Boudnik 26/Aug/13 20:54
        Resolved Resolved Closed Closed
        70d 1h 30m 1 Roman Shaposhnik 04/Nov/13 21:24

          People

          • Assignee:
            Ian Mordey
            Reporter:
            Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development