Hadoop Common
  1. Hadoop Common
  2. HADOOP-5232

preparing HadoopPatchQueueAdmin.sh,test-patch.sh scripts to run builds on hudson slaves.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      To modify hadoopPatchQueueAdmin.sh and test-patch.sh script to run patch builds on hudson slaves.

      1. hudsonPatchQueueAdmin.sh.patch
        3 kB
        Giridharan Kesavan
      2. nightly.patch
        8 kB
        Giridharan Kesavan
      3. nightly.patch
        8 kB
        Giridharan Kesavan
      4. nightly.patch
        6 kB
        Giridharan Kesavan
      5. test.patch
        2 kB
        Giridharan Kesavan
      6. test.patch
        2 kB
        Giridharan Kesavan
      7. test-patch.sh.patch
        2 kB
        Giridharan Kesavan

        Activity

        Hide
        Giridharan Kesavan added a comment -

        Here is the proposal for running parallel patch builds using hudson slaves.

        Plan:
        hudson.zones will continue to parse all the jira email traffic.
        hudson.zones will continue to run the patchAdmin job.
        hudson patch jobs will be created per slave basis. [rest assured that the patch runs only on that slave]
        ie. Whenever we plan to add a new hudson slave machine we need to create/add new job hadoop-patch job inhudson
        explicitly for that slave.

        slave/master communication happens through parametrized builds and posting values through url.
        And it seems this parametrized builds are supported since hudson version - 1.278
        We have to upgrade our hudson.zones-1.268 to run hudson 1.278 to utilize this feature.

        Apart from this we have to modify the way the patch admin calls the patch builds to use parametrized builds.
        Also the test-patch.sh script in the src/test/bin had to be modified to accomodate hudson slave patch builds.

        Finally the patchqueue html page will show up the currentpatch build's on different slaves at any point in time.

        Show
        Giridharan Kesavan added a comment - Here is the proposal for running parallel patch builds using hudson slaves. Plan: hudson.zones will continue to parse all the jira email traffic. hudson.zones will continue to run the patchAdmin job. hudson patch jobs will be created per slave basis. [rest assured that the patch runs only on that slave] ie. Whenever we plan to add a new hudson slave machine we need to create/add new job hadoop-patch job inhudson explicitly for that slave. slave/master communication happens through parametrized builds and posting values through url. And it seems this parametrized builds are supported since hudson version - 1.278 We have to upgrade our hudson.zones-1.268 to run hudson 1.278 to utilize this feature. Apart from this we have to modify the way the patch admin calls the patch builds to use parametrized builds. Also the test-patch.sh script in the src/test/bin had to be modified to accomodate hudson slave patch builds. Finally the patchqueue html page will show up the currentpatch build's on different slaves at any point in time.
        Hide
        Nigel Daley added a comment -

        Looks good. How do you plan to remove the 'current' directory from the patch queue once a build completes? Do you have a patch?

        Show
        Nigel Daley added a comment - Looks good. How do you plan to remove the 'current' directory from the patch queue once a build completes? Do you have a patch?
        Hide
        Giridharan Kesavan added a comment -

        Im thinking of having another parametrized hudson job just to do the cleanup of the current folder;
        This will be done by the test-patch.sh script which would call the hudson job as part of the cleanupAndExit function.

        Show
        Giridharan Kesavan added a comment - Im thinking of having another parametrized hudson job just to do the cleanup of the current folder; This will be done by the test-patch.sh script which would call the hudson job as part of the cleanupAndExit function.
        Hide
        Nigel Daley added a comment -

        Interesting idea. I suppose you could call Hadoop-Patch-Admin with a parameter. Then you'd get the cleanup done and the next patch kicked off right away.

        Show
        Nigel Daley added a comment - Interesting idea. I suppose you could call Hadoop-Patch-Admin with a parameter. Then you'd get the cleanup done and the next patch kicked off right away.
        Hide
        Giridharan Kesavan added a comment -

        I agree with your idea; I would make the changes appropriately and upload the patch..tnx!

        Show
        Giridharan Kesavan added a comment - I agree with your idea; I would make the changes appropriately and upload the patch..tnx!
        Hide
        Giridharan Kesavan added a comment -

        patch for test-patch.sh and patch for hudsonPatchQueueAdmin.sh attached to exuecute patch build on hudson slaves..

        Show
        Giridharan Kesavan added a comment - patch for test-patch.sh and patch for hudsonPatchQueueAdmin.sh attached to exuecute patch build on hudson slaves..
        Hide
        Nigel Daley added a comment -

        Thanks Giri.

        Comments on hudsonPatchQueueAdmin.sh.patch:

        1. why is $3 used instead of assigning to a variable name?
        2. set -x shouldn't be set in this script as it will leak the build token secret (which I think is $3) to the console
        3. where is BUILD_SERVERS defined? CALLEDBY?
        4. why use a directory to store defectNum ($CURRENT_DIR/defectNum)? – instead just keep defect numbers in a single file
        5. please preserve 2 space indents
        6. removed un-used lines (like #TRIGGER_BUILD_URL=$3)

        Comments on test-patch.sh.patch:

        1. where's the 18th parameter?
        2. give more info on why in this echo: "Calling the Patch Queue Admin"

        Can you provide examples of how these scripts are now called? (don't leak the secret build token).

        Show
        Nigel Daley added a comment - Thanks Giri. Comments on hudsonPatchQueueAdmin.sh.patch: why is $3 used instead of assigning to a variable name? set -x shouldn't be set in this script as it will leak the build token secret (which I think is $3) to the console where is BUILD_SERVERS defined? CALLEDBY? why use a directory to store defectNum ($CURRENT_DIR/defectNum)? – instead just keep defect numbers in a single file please preserve 2 space indents removed un-used lines (like #TRIGGER_BUILD_URL=$3) Comments on test-patch.sh.patch: where's the 18th parameter? give more info on why in this echo: "Calling the Patch Queue Admin" Can you provide examples of how these scripts are now called? (don't leak the secret build token).
        Hide
        Giridharan Kesavan added a comment - - edited

        I 've addressed all your comments from Nigel and submitting an improved version of the patch here...

        We have two patch here. one is called the nightly.patch which we should apply in the nightly folder
        and the second one called test.patch which we should apply on the hadoop base folder.

        All the comments mentioned by Nigel are addressed by these two patches. We no more have a folder for queues. we have a simple txt file which acts as a patch queue.

        Please Review.

        Here are the example cmd for patch-admin job and the patch build job.
        ###################################################
        PATCH_ADMIN_BUILD

        CALLER IS THE HUDSON PARAMATERISED VARIABLE and HAS ITS DEFAULT VALUE SET TO USER

        CMD:
        export BUILD_SERVERS="<SERVER2 HOSTNAME> <SERVER2 HOSTNAME>"
        CALLEDBY=$

        {CALLER}

        export CALLEDBY

        cd $workspace\nightly
        sh hudsonPatchQueueAdmin.sh HADOOP <patch_queue_dir> </buildWithParameters?token=<token> <PATH_TO_CURL>

        ###########################################

        PATCH BUILD:
        DEFECTNUM IS THE HUDSON PARAMATERISED VARIABLE and HAS ITS DEFAULT VALUE SET TO ""

        cmd:
        export defect=$

        {DEFECTNUM}

        export PATCH_ADMIN_URL=http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-Admin/buildWithParameters?token=<token>

        other options remaining the same we would be modifying the value for trigger.url
        -Dtrigger.url=http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/buildWithParameters?token=<token>
        and we would add one more variable for curl.
        -Dcurl.cmd=<path to curl>

        #############################################################

        Show
        Giridharan Kesavan added a comment - - edited I 've addressed all your comments from Nigel and submitting an improved version of the patch here... We have two patch here. one is called the nightly.patch which we should apply in the nightly folder and the second one called test.patch which we should apply on the hadoop base folder. All the comments mentioned by Nigel are addressed by these two patches. We no more have a folder for queues. we have a simple txt file which acts as a patch queue. Please Review. Here are the example cmd for patch-admin job and the patch build job. ################################################### PATCH_ADMIN_BUILD CALLER IS THE HUDSON PARAMATERISED VARIABLE and HAS ITS DEFAULT VALUE SET TO USER CMD: export BUILD_SERVERS="<SERVER2 HOSTNAME> <SERVER2 HOSTNAME>" CALLEDBY=$ {CALLER} export CALLEDBY cd $workspace\nightly sh hudsonPatchQueueAdmin.sh HADOOP <patch_queue_dir> </buildWithParameters?token=<token> <PATH_TO_CURL> ########################################### PATCH BUILD: DEFECTNUM IS THE HUDSON PARAMATERISED VARIABLE and HAS ITS DEFAULT VALUE SET TO "" cmd: export defect=$ {DEFECTNUM} export PATCH_ADMIN_URL= http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-Admin/buildWithParameters?token= <token> other options remaining the same we would be modifying the value for trigger.url -Dtrigger.url= http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/buildWithParameters?token= <token> and we would add one more variable for curl. -Dcurl.cmd=<path to curl> #############################################################
        Hide
        Nigel Daley added a comment -

        Looks good. More comments:

        hudsonPatchQueueAdmin.sh

        1. comment "The complete URL to trigger a build" is now wrong
        2. if CALLEDBY... needs some comments
        3. $CUR should be $CURL
        4. ...DEFECTNUM=`cat $CURRENT_PATCH` should be $...DEFECTNUM=$defect
        Show
        Nigel Daley added a comment - Looks good. More comments: hudsonPatchQueueAdmin.sh comment "The complete URL to trigger a build" is now wrong if CALLEDBY... needs some comments $CUR should be $CURL ...DEFECTNUM=`cat $CURRENT_PATCH` should be $...DEFECTNUM=$defect
        Hide
        Giridharan Kesavan added a comment -

        updated the patch as per comments...
        Added more functionality to the email processing script -processHadoopPatchEmail.sh and hudsonPatchQueueAdmin.sh

        • email processing scirpt checks for duplicate defect in the patch queue before adding defects to the queue
        • hudsonPatchQueueAdmin.sh doesnt require any env variables. It uses getopts to parse the cmd line arguments.

        example

         
        sh hudsonPatchQueueAdmin.sh -p<PROJECT> -s'SERVER1 SERVER2' -c<CALLING SERVER HOSTNAME> -t<PATH TO CURL> -u</buildWithParameters?token=<token>> -q<path to the patch queue>
        
        To enable execute the script in debug mode use -v along with other args.
        
        To display the script usage use the -h option.
        sh hudsonPatchQueueAdmin.sh -h 
        Show
        Giridharan Kesavan added a comment - updated the patch as per comments... Added more functionality to the email processing script -processHadoopPatchEmail.sh and hudsonPatchQueueAdmin.sh email processing scirpt checks for duplicate defect in the patch queue before adding defects to the queue hudsonPatchQueueAdmin.sh doesnt require any env variables. It uses getopts to parse the cmd line arguments. example sh hudsonPatchQueueAdmin.sh -p<PROJECT> -s'SERVER1 SERVER2' -c<CALLING SERVER HOSTNAME> -t<PATH TO CURL> -u</buildWithParameters?token=<token>> -q<path to the patch queue> To enable execute the script in debug mode use -v along with other args. To display the script usage use the -h option. sh hudsonPatchQueueAdmin.sh -h
        Hide
        Nigel Daley added a comment -

        +1

        Show
        Nigel Daley added a comment - +1
        Hide
        Giridharan Kesavan added a comment -

        fixed a typo in the nightly patch.

        Show
        Giridharan Kesavan added a comment - fixed a typo in the nightly patch.
        Hide
        Giridharan Kesavan added a comment -

        with the latest test.patch defect # is getting passed as an arg to the test-patch.sh script.

        Show
        Giridharan Kesavan added a comment - with the latest test.patch defect # is getting passed as an arg to the test-patch.sh script.
        Hide
        Nigel Daley added a comment -

        I just committed this. Thanks Giri.

        Show
        Nigel Daley added a comment - I just committed this. Thanks Giri.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #766 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/766/)
        . Enable patch testing to occur on more than one host. Contributed by Giri Kesavan.
        . Enable patch testing to occur on more than one host. Contributed by Giri Kesavan.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #766 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/766/ ) . Enable patch testing to occur on more than one host. Contributed by Giri Kesavan. . Enable patch testing to occur on more than one host. Contributed by Giri Kesavan.

          People

          • Assignee:
            Giridharan Kesavan
            Reporter:
            Giridharan Kesavan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development