Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1501

come up with a way to use Jenkins plugins/Groovy console to create jobs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: build
    • Labels:
      None

      Description

      The idea here is simple: we're now migrating to the new style of build infrastructure that is 100% Docker centric. This will make our Jenkins jobs highly regular and subject to automatic creation based on a template.

      At this point, I don't know of an easy way to automate this and would appreciate if somebody can research it.

      The pipeline I imagine would be this: there will be a meta-job on Jenkins running and creating other jobs based on the state of a template in our source code repo.

        Activity

        Hide
        gkesavan Giridharan Kesavan added a comment -

        Roman Shaposhnik jenkins has this yaml plugin , which could be used for configuring jobs.

        https://wiki.jenkins-ci.org/display/JENKINS/YAML+Project+Plugin

        Show
        gkesavan Giridharan Kesavan added a comment - Roman Shaposhnik jenkins has this yaml plugin , which could be used for configuring jobs. https://wiki.jenkins-ci.org/display/JENKINS/YAML+Project+Plugin
        Hide
        kaiyzen Nate DAmico added a comment -
        Show
        kaiyzen Nate DAmico added a comment - Looking towards the Job DSL plugin would be an approach: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin http://www.slideshare.net/quidryan/configuration-as-code
        Hide
        jayunit100 jay vyas added a comment - - edited

        the job dsl thing seems cool . but... my feelings are , with exception of things like release management and reporting — in general, with immutable infra practices encoueraged by things like vagrant and docker, much of what build servers used to be required for — is now provided in the local dev infra anyways.

        so id suggest we move as much as possible out of the jenkins servers... i could be missing context here though

        Show
        jayunit100 jay vyas added a comment - - edited the job dsl thing seems cool . but... my feelings are , with exception of things like release management and reporting — in general, with immutable infra practices encoueraged by things like vagrant and docker, much of what build servers used to be required for — is now provided in the local dev infra anyways. so id suggest we move as much as possible out of the jenkins servers... i could be missing context here though
        Hide
        rvs Roman Shaposhnik added a comment -

        Giridharan Kesavan and Nate DAmico thank you guys for feedback! I'm seriously looking into Job DSL Plugin now!

        jay vyas the idea with Job DSL plugin is to do exactly what you're suggesting: have close to 0 amount of state in Jenkins configuration itself. Basically all state of Jenkins will be kept in Bigtop source code and would be expected to be able to completely boostrap from a single seed Job reading the DSL. At least that's the hope.

        Show
        rvs Roman Shaposhnik added a comment - Giridharan Kesavan and Nate DAmico thank you guys for feedback! I'm seriously looking into Job DSL Plugin now! jay vyas the idea with Job DSL plugin is to do exactly what you're suggesting: have close to 0 amount of state in Jenkins configuration itself. Basically all state of Jenkins will be kept in Bigtop source code and would be expected to be able to completely boostrap from a single seed Job reading the DSL. At least that's the hope.
        Hide
        rvs Roman Shaposhnik added a comment -

        Attaching a very rough first cut at what I have in mind. Please, please comment.

        Show
        rvs Roman Shaposhnik added a comment - Attaching a very rough first cut at what I have in mind. Please, please comment.
        Hide
        rvs Roman Shaposhnik added a comment -

        Btw, for those who would like to see what running this script would produce on our Jenkins, here's a tab that was generated: http://bigtop01.cloudera.org:8080/view/Bootstrap/

        Nothing in this view was created manually.

        Show
        rvs Roman Shaposhnik added a comment - Btw, for those who would like to see what running this script would produce on our Jenkins, here's a tab that was generated: http://bigtop01.cloudera.org:8080/view/Bootstrap/ Nothing in this view was created manually.
        Hide
        cos Konstantin Boudnik added a comment -

        Roman Shaposhnik, do you think it is ready for the review/commit?

        Show
        cos Konstantin Boudnik added a comment - Roman Shaposhnik , do you think it is ready for the review/commit?
        Hide
        rvs Roman Shaposhnik added a comment -

        Almost. I need to polish the Package testing side of things a little bit. Give me a weekend.

        Show
        rvs Roman Shaposhnik added a comment - Almost. I need to polish the Package testing side of things a little bit. Give me a weekend.
        Hide
        cos Konstantin Boudnik added a comment -

        off-hand comment: we don't have Whirr anymore. Keep on reviewing

        Show
        cos Konstantin Boudnik added a comment - off-hand comment: we don't have Whirr anymore. Keep on reviewing
        Hide
        cos Konstantin Boudnik added a comment -

        and I think there's a way to read the list of components from BOM file as in packages.gradle:566

        Show
        cos Konstantin Boudnik added a comment - and I think there's a way to read the list of components from BOM file as in packages.gradle:566
        Hide
        cos Konstantin Boudnik added a comment -

        Besides of the need to remove Whirr thing I think the patch is good - let's commit it, so we can improve on it gradually. Especially considering how easy it is going to be to regenerate the jobs...

        Show
        cos Konstantin Boudnik added a comment - Besides of the need to remove Whirr thing I think the patch is good - let's commit it, so we can improve on it gradually. Especially considering how easy it is going to be to regenerate the jobs...
        Hide
        cos Konstantin Boudnik added a comment -

        That apparently has been committed. Not sure if by a mistake or else. Shall the ticket be closed or the potentially accidental commit shall be reverted?

        Show
        cos Konstantin Boudnik added a comment - That apparently has been committed. Not sure if by a mistake or else. Shall the ticket be closed or the potentially accidental commit shall be reverted?
        Hide
        rvs Roman Shaposhnik added a comment -

        I thought the idea was to commit and iterate? Let me know if you'd rather see it backed out.

        Show
        rvs Roman Shaposhnik added a comment - I thought the idea was to commit and iterate? Let me know if you'd rather see it backed out.
        Hide
        cos Konstantin Boudnik added a comment -

        Yup, that's what the idea. But I thought the iteration will be happening on separate tickets, as per commit per ticket?

        Show
        cos Konstantin Boudnik added a comment - Yup, that's what the idea. But I thought the iteration will be happening on separate tickets, as per commit per ticket?
        Hide
        cos Konstantin Boudnik added a comment -

        So, shall we resolve this and move the future progress elsewhere?

        Show
        cos Konstantin Boudnik added a comment - So, shall we resolve this and move the future progress elsewhere?
        Hide
        cos Konstantin Boudnik added a comment -

        I am resolving this because the main patch was committed. Let's continue the work on subsequent JIRAs. Thanks Roman!

        Show
        cos Konstantin Boudnik added a comment - I am resolving this because the main patch was committed. Let's continue the work on subsequent JIRAs. Thanks Roman!

          People

          • Assignee:
            rvs Roman Shaposhnik
            Reporter:
            rvs Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development