Infrastructure
  1. Infrastructure
  2. INFRA-4343

setup to build incubator Cordova weinre with buildbot

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: Initial Clearing
    • Component/s: Buildbot
    • Labels:
      None

      Description

      I'm getting ready to make use of Apache's build infrastructure to perform builds for the weinre product that's part of the Adobe Cordova incubator.

      What are the next steps?

        Activity

        Hide
        Patrick Mueller added a comment -
        It's been so long since I've played with this, I'm not even sure what else needs to be done. I've survived without the CI bits until now, I think I can continue without them for a while more. I'll open a new bug if I ever get a desire to play with this again.
        Show
        Patrick Mueller added a comment - It's been so long since I've played with this, I'm not even sure what else needs to be done. I've survived without the CI bits until now, I think I can continue without them for a while more. I'll open a new bug if I ever get a desire to play with this again.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Ping, what work is outstanding here?
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Ping, what work is outstanding here?
        Hide
        Tony Stevenson added a comment -
        Transitioning to waiting for Infra
        Show
        Tony Stevenson added a comment - Transitioning to waiting for Infra
        Hide
        Patrick Mueller added a comment -
        woops, also forgot to mention that I was watching #asftest to see a message that the build would be kicked off, but it never got posted. Any way of making that happen? Guess I'd like to see a message when it starts, and a message when it ends. If the messages contain links to the build reports, awesome, but if not, survivable.
        Show
        Patrick Mueller added a comment - woops, also forgot to mention that I was watching #asftest to see a message that the build would be kicked off, but it never got posted. Any way of making that happen? Guess I'd like to see a message when it starts, and a message when it ends. If the messages contain links to the build reports, awesome, but if not, survivable.
        Hide
        Patrick Mueller added a comment -
        So, awesome news, the build did in fact run triggered off the commit (I assume). Here's the build that occurred:

            http://ci.apache.org/builders/cordova-weinre/builds/9

        I'm now tweaking the cordova.conf script for some changes due to recent project changes.

        Still don't know what to do with archives being generated by this build-bot.
        Show
        Patrick Mueller added a comment - So, awesome news, the build did in fact run triggered off the commit (I assume). Here's the build that occurred:      http://ci.apache.org/builders/cordova-weinre/builds/9 I'm now tweaking the cordova.conf script for some changes due to recent project changes. Still don't know what to do with archives being generated by this build-bot.
        Hide
        Patrick Mueller added a comment -
        One of the missing pieces I don't quite understand is where I'm supposed to put my build artifacts. I have a place for them on my people account:

            http://people.apache.org/~pmuellr/weinre/builds/

        Wasn't clear to me if there was already a "ci build archive" location set up, auto-managed, whatever.

        I'm fine copying archives over to my people account, but would need some guidance on how to do that. I don't want to have to install .ssh goop on a ci server so I can password-free scp the files over, and obviously don't want to store passwords on a ci server.

        If folks copy build artifacts from ci servers to other apache machines, what mechanism are they using?
        Show
        Patrick Mueller added a comment - One of the missing pieces I don't quite understand is where I'm supposed to put my build artifacts. I have a place for them on my people account:      http://people.apache.org/~pmuellr/weinre/builds/ Wasn't clear to me if there was already a "ci build archive" location set up, auto-managed, whatever. I'm fine copying archives over to my people account, but would need some guidance on how to do that. I don't want to have to install .ssh goop on a ci server so I can password-free scp the files over, and obviously don't want to store passwords on a ci server. If folks copy build artifacts from ci servers to other apache machines, what mechanism are they using?
        Hide
        Patrick Mueller added a comment -
        Haven't been able to re-look at this till today, because I finally pushed a commit to apache's git master branch:

            https://git-wip-us.apache.org/repos/asf?p=incubator-cordova-weinre.git;a=commit;h=7ccffd765d7b78c9cb32b8f96a7a8c966b00fd3b

        I've swizzled the build, wouldn't surprise me if the build fails, but that's ok. I think I can drive changes to the build, at this point, if I can just get the build kicked off on a per-commit basis.
        Show
        Patrick Mueller added a comment - Haven't been able to re-look at this till today, because I finally pushed a commit to apache's git master branch:      https://git-wip-us.apache.org/repos/asf?p=incubator-cordova-weinre.git;a=commit;h=7ccffd765d7b78c9cb32b8f96a7a8c966b00fd3b I've swizzled the build, wouldn't surprise me if the build fails, but that's ok. I think I can drive changes to the build, at this point, if I can just get the build kicked off on a per-commit basis.
        Hide
        Gavin added a comment -
        [note that Patrick is away for a week before he can try this again]
        Show
        Gavin added a comment - [note that Patrick is away for a week before he can try this again]
        Hide
        Gavin added a comment -
        Please try a commit again in master and lets see if it triggers, I missed a config option on the git server.
        Show
        Gavin added a comment - Please try a commit again in master and lets see if it triggers, I missed a config option on the git server.
        Hide
        Patrick Mueller added a comment -
        Today I updated my master branch with a huge # of changes. It's the first update to the master branch since the ci build was set up. Unfortunately, the auto-build did not get triggered. Waited over an hour. Figured it should have run by then.

        No big worries, figured out I could force the build via #asftest messages to cordova-bot, which worked out AWESOMELY. neat!

        Now, gotta get the archives in .../weinre.build/out/archives copied somewhere. Per previous comment, keeping 10 of them should be fine.

        I may want to add another build based on a new (non-existent) branch called "develop". Turns out I had a bug in my build script, so I had to commit a fix to master to get the build running. I'd prefer for those kind of commits to not have to occur on master - having a develop branch would be nice to fix issues like that.

        Assuming I can figure out how to change the config, and checkin changes to it, do I have to inform someone/something that a new config is available to be re-read?
        Show
        Patrick Mueller added a comment - Today I updated my master branch with a huge # of changes. It's the first update to the master branch since the ci build was set up. Unfortunately, the auto-build did not get triggered. Waited over an hour. Figured it should have run by then. No big worries, figured out I could force the build via #asftest messages to cordova-bot, which worked out AWESOMELY. neat! Now, gotta get the archives in .../weinre.build/out/archives copied somewhere. Per previous comment, keeping 10 of them should be fine. I may want to add another build based on a new (non-existent) branch called "develop". Turns out I had a bug in my build script, so I had to commit a fix to master to get the build running. I'd prefer for those kind of commits to not have to occur on master - having a develop branch would be nice to fix issues like that. Assuming I can figure out how to change the config, and checkin changes to it, do I have to inform someone/something that a new config is available to be re-read?
        Hide
        Patrick Mueller added a comment -
        ah, makes sense, being "ci" and all.

        yes, let's create an area, and keep 10 builds for now.

        thanks much!
        Show
        Patrick Mueller added a comment - ah, makes sense, being "ci" and all. yes, let's create an area, and keep 10 builds for now. thanks much!
        Hide
        Gavin added a comment -
        Hi Patrick, there are lots of documentation and your mentors to help you answer those questions.
        It's too much for me to answer on this issue.

        What I'm here for, and what this issue is about is getting the build to pass, and optionally to provide a permanant location for your unofficial builds. (ci.apache.org/projects/cordova/weinre/snapshots/ for example would be a permanent location you can refer to in your site docs etc.)

        The CI machines do not build official releases nor do they house them. Official releases are releases that have to be voted on by the podling and also the IPMC before being signed and released to an official downloads area and distributed to our mirror system - none of which is a buildbot task.

        The builds I have done so far is configured to trigger when a commit is made to your git master branch of the weinre build.

        What I think is left to do here is create you an area for your unnoficial builds and configure it to keep say the last 10 or 20 builds, depending on frequency.
        Let me know if you want me to go ahead with that. And anything else you think needs doing here. The rest, please talk to your mentors and read various release guides such as http://incubator.apache.org/guides/releasemanagement.html#intro would be a good place to start.
        Show
        Gavin added a comment - Hi Patrick, there are lots of documentation and your mentors to help you answer those questions. It's too much for me to answer on this issue. What I'm here for, and what this issue is about is getting the build to pass, and optionally to provide a permanant location for your unofficial builds. (ci.apache.org/projects/cordova/weinre/snapshots/ for example would be a permanent location you can refer to in your site docs etc.) The CI machines do not build official releases nor do they house them. Official releases are releases that have to be voted on by the podling and also the IPMC before being signed and released to an official downloads area and distributed to our mirror system - none of which is a buildbot task. The builds I have done so far is configured to trigger when a commit is made to your git master branch of the weinre build. What I think is left to do here is create you an area for your unnoficial builds and configure it to keep say the last 10 or 20 builds, depending on frequency. Let me know if you want me to go ahead with that. And anything else you think needs doing here. The rest, please talk to your mentors and read various release guides such as http://incubator.apache.org/guides/releasemanagement.html#intro would be a good place to start.
        Hide
        Patrick Mueller added a comment -
        Great!

        So, yeah, I guess the next step is getting the archives built moved into an accessible location, and you found them! weinre.build/out/archives

        BUT - DON'T DO THAT RIGHT NOW (see 4. and 5. below)

        There's a couple things I need to figure out:

        1) are there standards for how files in archives are laid out?
        2) should I create .tgz's/etc in addition to .zip's?
        3) howzabout sha signature files? do I create these?
        4) given you statement about daily builds, I'd guess these are not "official" builds, right? In fact, the archive's source files do not contain Apache license info (my next (HUGE) commit to master will fix this)
        5) since these builds aren't kosher (missing licensing), no sense in actually moving the archives to a download directory just yet
        6) not sure that a daily build is warranted in any case, since there typically aren't commits every day
        7) how do I clean out these transient builds?
        8) presumably daily builds are not kept forever, but there are non-final releases that are; eg, "rc" releases. How does that work?
        9) how do I do official builds?
        10) I'd guess there is some special procedure for doing "official builds", staging them, and them making them available marked as "official"?

        I think I would be happy with a process where a weinre build is kicked off for every commit to master. The build time should drop to ~10 seconds in the next (HUGE) commit.

        This would just be a couple builds a week, I would think. I do all my work on other branches and commit to master only when I'm completely finished with a feature/bug-fix.

        But do need to figure out how to do "official" builds as well.
        Show
        Patrick Mueller added a comment - Great! So, yeah, I guess the next step is getting the archives built moved into an accessible location, and you found them! weinre.build/out/archives BUT - DON'T DO THAT RIGHT NOW (see 4. and 5. below) There's a couple things I need to figure out: 1) are there standards for how files in archives are laid out? 2) should I create .tgz's/etc in addition to .zip's? 3) howzabout sha signature files? do I create these? 4) given you statement about daily builds, I'd guess these are not "official" builds, right? In fact, the archive's source files do not contain Apache license info (my next (HUGE) commit to master will fix this) 5) since these builds aren't kosher (missing licensing), no sense in actually moving the archives to a download directory just yet 6) not sure that a daily build is warranted in any case, since there typically aren't commits every day 7) how do I clean out these transient builds? 8) presumably daily builds are not kept forever, but there are non-final releases that are; eg, "rc" releases. How does that work? 9) how do I do official builds? 10) I'd guess there is some special procedure for doing "official builds", staging them, and them making them available marked as "official"? I think I would be happy with a process where a weinre build is kicked off for every commit to master. The build time should drop to ~10 seconds in the next (HUGE) commit. This would just be a couple builds a week, I would think. I do all my work on other branches and commit to master only when I'm completely finished with a feature/bug-fix. But do need to figure out how to do "official" builds as well.
        Hide
        Gavin added a comment -
        Thanks, I thought something was amiss, I had already installed node via the instructions mentioned.
        Now I've installed NPM too.

        The build now passes:

        http://ci.apache.org/builders/cordova-weinre/builds/5/

        this is enessentially performing 'ant clean && ant'

        Whats next? Do you need the weinre.build/out/archives uploading for availability?

        Currently they end up with names like weinre-src-1.6.1build-2012-02-09--20-16-22.zip .

        If you are going to want these zips available, I'd suggest we only perform the build daily.

        We generally upload to the ci.apache.org/project/ URL space.

        Some examples are available from http://ci.apache.org/projects/ page.

        Such as:

        http://ci.apache.org/projects/ofbiz/snapshots/index.html

        http://ci.apache.org/projects/jmeter/nightlies/index.html

        http://ci.apache.org/projects/subversion/nightlies/index.html

        You'll notice the pages are built to match their own website styles, I would
        do the same for your nightlies area if this is the way you want to go.

        Anyway, whats next?

        Show
        Gavin added a comment - Thanks, I thought something was amiss, I had already installed node via the instructions mentioned. Now I've installed NPM too. The build now passes: http://ci.apache.org/builders/cordova-weinre/builds/5/ this is enessentially performing 'ant clean && ant' Whats next? Do you need the weinre.build/out/archives uploading for availability? Currently they end up with names like weinre-src-1.6.1build-2012-02-09--20-16-22.zip . If you are going to want these zips available, I'd suggest we only perform the build daily. We generally upload to the ci.apache.org/project/ URL space. Some examples are available from http://ci.apache.org/projects/ page. Such as: http://ci.apache.org/projects/ofbiz/snapshots/index.html http://ci.apache.org/projects/jmeter/nightlies/index.html http://ci.apache.org/projects/subversion/nightlies/index.html You'll notice the pages are built to match their own website styles, I would do the same for your nightlies area if this is the way you want to go. Anyway, whats next?
        Hide
        Patrick Mueller added a comment -
        I just tried installing node via the linked to instructions, on a 32-bit ubuntu 10.04.3 desktop image; worked fine, installed nodejs 0.6.10 .

        Unfortunately, this did not also install npm, which the 'build-from-scratch' and 'install from nodejs Mac package on nodejs site' do.

        To get npm installed, you also need to install "npm" - version 1.1.0 is the one that it picked up, which is fine.

        Note I installed via Synaptic Package Manager rather than the cmd line, but assume cmd line will work as well

            sudo apt-get install npm

            
        Show
        Patrick Mueller added a comment - I just tried installing node via the linked to instructions, on a 32-bit ubuntu 10.04.3 desktop image; worked fine, installed nodejs 0.6.10 . Unfortunately, this did not also install npm, which the 'build-from-scratch' and 'install from nodejs Mac package on nodejs site' do. To get npm installed, you also need to install "npm" - version 1.1.0 is the one that it picked up, which is fine. Note I installed via Synaptic Package Manager rather than the cmd line, but assume cmd line will work as well     sudo apt-get install npm     
        Hide
        Patrick Mueller added a comment -
        awesome thanks.

        Next step is that the buildslave20 machine (guess that's it's name) needs to have node installed on it somewhere. A stable version, >= 0.6.4. Presumably on the path. If it's not on the path, then I at least need to know where it's installed.

        Installing node [1] should also install npm, which will get us to the next step of the build.

        [1] https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
        Show
        Patrick Mueller added a comment - awesome thanks. Next step is that the buildslave20 machine (guess that's it's name) needs to have node installed on it somewhere. A stable version, >= 0.6.4. Presumably on the path. If it's not on the path, then I at least need to know where it's installed. Installing node [1] should also install npm, which will get us to the next step of the build. [1] https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
        Hide
        Gavin added a comment -
        Ok I've got a basic build set up.

        Various views:

        http://ci.apache.org/builders/cordova-weinre
        http://ci.apache.org/waterfall?show=cordova-weinre
        http://ci.apache.org/builders/cordova-weinre/builds/3
        http://ci.apache.org/builders/cordova-weinre/builds/3/steps/compile_1/logs/stdio

        That last link we are interested in currently as it shows the current failure.

        "...weinre.build/get-vendor.xml:117: Execute failed: java.io.IOException: Cannot run program "npm": error=2, No such file or directory..."

        Not sure if thats for you to fix or if I need to do something else, all dependencies are installed as far as I can tell.

        The buildbot config file for cordova is at :

        https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/cordova.conf

        you have commit rights to that file but be careful, as well as the build we are doing here that file also contains your website CMS config so steer clear of that part.
        Ask if you have any problems or I am happy to continue on with the editing of that file until you get the idea.

        Anyway, whats the next step to fixing the build?


        Show
        Gavin added a comment - Ok I've got a basic build set up. Various views: http://ci.apache.org/builders/cordova-weinre http://ci.apache.org/waterfall?show=cordova-weinre http://ci.apache.org/builders/cordova-weinre/builds/3 http://ci.apache.org/builders/cordova-weinre/builds/3/steps/compile_1/logs/stdio That last link we are interested in currently as it shows the current failure. "...weinre.build/get-vendor.xml:117: Execute failed: java.io.IOException: Cannot run program "npm": error=2, No such file or directory..." Not sure if thats for you to fix or if I need to do something else, all dependencies are installed as far as I can tell. The buildbot config file for cordova is at : https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/cordova.conf you have commit rights to that file but be careful, as well as the build we are doing here that file also contains your website CMS config so steer clear of that part. Ask if you have any problems or I am happy to continue on with the editing of that file until you get the idea. Anyway, whats the next step to fixing the build?
        Hide
        Patrick Mueller added a comment -
        The basic thing I want set up is this:

        - when there's an SCM commit, I'd like a build launched. I think this means I want at least two branches watched, and the build would check out the relevant branch that changed. The branches would be "master" and a new "develop" branch I'd set up as the staging area for "master".

        - the current build is an ant script [1]; eventually, that will be rewritten to a Make/jake/cake file

        - the outputs are currently written to a directory weinre.build/out/archives - will be three zip files; weinre.zip, weinre-src.zip, weinre-doc.zip. The file names are suffixed with version and date, like so: 1.6.1-2012-02-05--13-54-44; I'll need to read up on how Apache archives are suffixed like this - I think they typically do not include the date, which is fine.

        At the moment, you you try to run the build.xml file, it will die fairly quickly with an error. I expect to push changes to master within a week which will enable the build.xml to work again.

        [1] https://git-wip-us.apache.org/repos/asf?p=incubator-cordova-weinre.git;a=blob;f=weinre.build/build.xml
        Show
        Patrick Mueller added a comment - The basic thing I want set up is this: - when there's an SCM commit, I'd like a build launched. I think this means I want at least two branches watched, and the build would check out the relevant branch that changed. The branches would be "master" and a new "develop" branch I'd set up as the staging area for "master". - the current build is an ant script [1]; eventually, that will be rewritten to a Make/jake/cake file - the outputs are currently written to a directory weinre.build/out/archives - will be three zip files; weinre.zip, weinre-src.zip, weinre-doc.zip. The file names are suffixed with version and date, like so: 1.6.1-2012-02-05--13-54-44; I'll need to read up on how Apache archives are suffixed like this - I think they typically do not include the date, which is fine. At the moment, you you try to run the build.xml file, it will die fairly quickly with an error. I expect to push changes to master within a week which will enable the build.xml to work again. [1] https://git-wip-us.apache.org/repos/asf?p=incubator-cordova-weinre.git;a=blob;f=weinre.build/build.xml
        Hide
        Gavin added a comment -
        I can sort the commands into buildbot for the initial setup, just tell me what you normally run to perform a build.

        Once I have a basic config in place and working, you'll have access to it to tweak/add to it etc no problem.
        Show
        Gavin added a comment - I can sort the commands into buildbot for the initial setup, just tell me what you normally run to perform a build. Once I have a basic config in place and working, you'll have access to it to tweak/add to it etc no problem.
        Hide
        Patrick Mueller added a comment -
        Honestly, I have no idea what I'm doing here. You are correct about using the git-weinre repo as source, but it's not currently in a state where it will build. Running ant on the weinre.build/build.xml file should result in some errors, for the master branch.

        Guess I need to read up on BuildBot. Apache-specific info on BuildBot seemed pretty light [1]

        How much administration can I do myself, in terms of setting up new build commands to be executed, when they run, etc. And how do I do that?

        [1] http://ci.apache.org/buildbot.html
        Show
        Patrick Mueller added a comment - Honestly, I have no idea what I'm doing here. You are correct about using the git-weinre repo as source, but it's not currently in a state where it will build. Running ant on the weinre.build/build.xml file should result in some errors, for the master branch. Guess I need to read up on BuildBot. Apache-specific info on BuildBot seemed pretty light [1] How much administration can I do myself, in terms of setting up new build commands to be executed, when they run, etc. And how do I do that? [1] http://ci.apache.org/buildbot.html
        Hide
        Gavin added a comment -
        Ok will be setting this up tomorrow.

        I guess you want me to use the git-weinre repo as source. Have you got some basic build commands you want to be using so I can start testing it as I install? Thanks,.
        Show
        Gavin added a comment - Ok will be setting this up tomorrow. I guess you want me to use the git-weinre repo as source. Have you got some basic build commands you want to be using so I can start testing it as I install? Thanks,.
        Hide
        Patrick Mueller added a comment -
        ubuntu is fine. any 2.x python > 2.4 or so will be fine. If node isn't already installed, node 0.6.9 (latest stable) would be great, otherwise some >= 0.6.3 (npm is shipped with node starting with version 0.6.3).

        Just looked at the node install would go with apt-get or equivalent; instructions are here [1]. Not knowing much about apt, not sure how kosher it is to require a new repository. I did an install through the Synaptic GUI (after adding the repo), and looks like it went fine anyway.

        [1] https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
        Show
        Patrick Mueller added a comment - ubuntu is fine. any 2.x python > 2.4 or so will be fine. If node isn't already installed, node 0.6.9 (latest stable) would be great, otherwise some >= 0.6.3 (npm is shipped with node starting with version 0.6.3). Just looked at the node install would go with apt-get or equivalent; instructions are here [1]. Not knowing much about apt, not sure how kosher it is to require a new repository. I did an install through the Synaptic GUI (after adding the repo), and looks like it went fine anyway. [1] https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
        Hide
        Gavin added a comment -
        You dont specify which OS platform to use, so unless objections, I'll stick it on an Ubuntu slave.
        Show
        Gavin added a comment - You dont specify which OS platform to use, so unless objections, I'll stick it on an Ubuntu slave.
        Hide
        Patrick Mueller added a comment -
        One of the things I'm curious about is what software is available on the buildbots, for me to build my product with. Asking on the mailing list yielded no responses. So, figured I'd ask here.

        http://mail-archives.apache.org/mod_mbox/www-builds/201201.mbox/%3CCACAoQH1sRU2rjztzLixz%2BoLBev9_h6jNiN8dWTWGCm7Fjz4MVQ%40mail.gmail.com%3E

        There are a few things which I suspect are already available:

        - javac
        - ant

        others I'm not sure about:

        - python
        - node.js
        Show
        Patrick Mueller added a comment - One of the things I'm curious about is what software is available on the buildbots, for me to build my product with. Asking on the mailing list yielded no responses. So, figured I'd ask here. http://mail-archives.apache.org/mod_mbox/www-builds/201201.mbox/%3CCACAoQH1sRU2rjztzLixz%2BoLBev9_h6jNiN8dWTWGCm7Fjz4MVQ%40mail.gmail.com%3E There are a few things which I suspect are already available: - javac - ant others I'm not sure about: - python - node.js

          People

          • Assignee:
            Gavin
            Reporter:
            Patrick Mueller
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development