Bigtop
  1. Bigtop
  2. BIGTOP-1201

Enhance (gradleize) the build to ease development, deployment; abstract implementation

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:
      None

      Description

      As has been discussed on the multiple occasions different parts of the Bigtop framework aren't really well stitched together. It requires a certain level of understanding of the project or/and digging across different sources of documentation to be able to:

      • build Bigtop framework
      • build and install target artifacts (packages, jars)
      • prepared to develop or test targeted stack
      • enforce all per-requisites consistently
      • coordinate version updates
      • produce documentation
      • <add more here>

      There are isolated attempts to patch the flaw with small pieces of scripting band-aids. However, the problem requires more systematic approach, in my opinion. What we need to have is a declarative yet flexible build system that can

      • short to medium term: wrap isolated pieces of the framework and provide a single entry point to building, developing, deployment, and testing
      • long term: consolidate all bits of the framework in one comprehensive build management system

      An apparent requirements of the solution:

      • it needs to play well with JVM stack
      • it needs to be expressive and declarative to give us flexibility to incorporate a number of currently used frameworks (maven, make, puppet) in one focal point
      • don't limit our ability to replace (if needed) various bits of current framework with something more uniform.

      Behold, I propose Gradle.

      1. BIGTOP-1201.patch
        29 kB
        Konstantin Boudnik
      2. BIGTOP-1201.patch
        28 kB
        Konstantin Boudnik
      3. BIGTOP-1201.patch
        28 kB
        Konstantin Boudnik
      4. BIGTOP-1201.patch
        27 kB
        Konstantin Boudnik
      5. BIGTOP-1201.patch
        26 kB
        Konstantin Boudnik
      6. BIGTOP-1201.patch
        23 kB
        Konstantin Boudnik
      7. BIGTOP-1201.patch
        26 kB
        Konstantin Boudnik
      8. BIGTOP-1201.patch
        20 kB
        Konstantin Boudnik
      9. BIGTOP-1201.patch
        18 kB
        Konstantin Boudnik
      10. BIGTOP-1201.patch
        18 kB
        Konstantin Boudnik
      11. BIGTOP-1201.patch
        17 kB
        Konstantin Boudnik
      12. BIGTOP-1201.patch
        17 kB
        Konstantin Boudnik
      13. BIGTOP-1201.patch
        16 kB
        Konstantin Boudnik
      14. BIGTOP-1201.patch
        14 kB
        Konstantin Boudnik
      15. BIGTOP-1201.patch
        12 kB
        Konstantin Boudnik
      16. BIGTOP-1201.patch
        5 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Committed to the master. Thanks everyone for the reviews and all the help. I just realized that I was working on it on-n-off for almost three months. Wow

          Show
          Konstantin Boudnik added a comment - Committed to the master. Thanks everyone for the reviews and all the help. I just realized that I was working on it on-n-off for almost three months. Wow
          Hide
          jay vyas added a comment -

          thanks cos. after initial review i had removed your patch and thats why it was borked . ill test more later after i pull down the official commit. initial validations looked great.

          Can you file a follow up JIRA paves the path to unify the build, remove redundnant stuff in bigtop.mk .

          this is great work sir ! its also a good way for all bigtop folks to learn gradle. hope you;ll share some of the finer details about how it works in a blog post or mail to the dev list .

          Show
          jay vyas added a comment - thanks cos. after initial review i had removed your patch and thats why it was borked . ill test more later after i pull down the official commit. initial validations looked great. Can you file a follow up JIRA paves the path to unify the build, remove redundnant stuff in bigtop.mk . this is great work sir ! its also a good way for all bigtop folks to learn gradle. hope you;ll share some of the finer details about how it works in a blog post or mail to the dev list .
          Hide
          Konstantin Boudnik added a comment -

          Thanks Jay. Here're the answers:

          1. I am not sure what you're referring to, but I would be under an impression that it is a homogeneous build environment that doesn't require a user to go cross framework boundaries as you perform different activities; be it packages creation, deployment, testing, etc. I'd think having everything driven by Gradle can accomplish it.
          2. The purpose of the target is to build local yum repo, so the packages can later be installed using yum command. Actually, I need to change this because right now this target is created for each component, which doesn't make any sense, really. How exactly gradle yum failed? Can you be more specific?
          3. build.gradle lines 20-21 explains just that
            // All packaging logic is separated into its own build module
            apply from: 'packages.gradle'
            
          Show
          Konstantin Boudnik added a comment - Thanks Jay. Here're the answers: I am not sure what you're referring to, but I would be under an impression that it is a homogeneous build environment that doesn't require a user to go cross framework boundaries as you perform different activities; be it packages creation, deployment, testing, etc. I'd think having everything driven by Gradle can accomplish it. The purpose of the target is to build local yum repo, so the packages can later be installed using yum command. Actually, I need to change this because right now this target is created for each component, which doesn't make any sense, really. How exactly gradle yum failed? Can you be more specific? build.gradle lines 20-21 explains just that // All packaging logic is separated into its own build module apply from: 'packages.gradle'
          Hide
          jay vyas added a comment - - edited

          +1 for commit, looks good to me !

          three last questions though :

          1) what is the "single push button build" ?

          2) I tried "gradle yum" and it failed, even though there seems to be a task called "yum" . what is the "yum " task ?

             task "yum" (dependsOn: tasks.findAll { alltask -> alltask.name.endsWith("-yum")}*.name) << { }
          

          3) How does packages.gradle relate to build.gradle?

          Show
          jay vyas added a comment - - edited +1 for commit, looks good to me ! three last questions though : 1) what is the "single push button build" ? 2) I tried "gradle yum" and it failed, even though there seems to be a task called "yum" . what is the "yum " task ? task "yum" (dependsOn: tasks.findAll { alltask -> alltask.name.endsWith("-yum")}*.name) << { } 3) How does packages.gradle relate to build.gradle?
          Hide
          Konstantin Boudnik added a comment -

          so, no objections on your side or you still need more time to pock around?

          Show
          Konstantin Boudnik added a comment - so, no objections on your side or you still need more time to pock around?
          Hide
          jay vyas added a comment - - edited

          +1, looks good !

          I applied the patch :

              patch -p0 < BIGTOP-1201.patch 
          

          And then ran

          gradle mahout-rpm
          

          Which resulted in , after a few minutes:

          .
          ├── BUILD
          ├── BUILDROOT
          ├── RPMS
          │   └── noarch
          │       ├── mahout-0.9-1.el6.noarch.rpm (59 MB)
          │       └── mahout-doc-0.9-1.el6.noarch.rpm
          ├── SOURCES
          │   ├── bigtop.bom
          │   └── init.d.tmpl
          ├── SPECS
          └── SRPMS
              └── mahout-0.9-1.el6.src.rpm
          

          So as an initial stab of testing it looks like it is working for packaging etc.

          Show
          jay vyas added a comment - - edited +1, looks good ! I applied the patch : patch -p0 < BIGTOP-1201.patch And then ran gradle mahout-rpm Which resulted in , after a few minutes: . ├── BUILD ├── BUILDROOT ├── RPMS │   └── noarch │   ├── mahout-0.9-1.el6.noarch.rpm (59 MB) │   └── mahout-doc-0.9-1.el6.noarch.rpm ├── SOURCES │   ├── bigtop.bom │   └── init.d.tmpl ├── SPECS └── SRPMS └── mahout-0.9-1.el6.src.rpm So as an initial stab of testing it looks like it is working for packaging etc.
          Hide
          jay vyas added a comment -

          i have a spare cycle right now ill apply the patch now and do some more playing with it and ping back.

          Show
          jay vyas added a comment - i have a spare cycle right now ill apply the patch now and do some more playing with it and ping back.
          Hide
          Konstantin Boudnik added a comment - - edited

          if there are no objections to the latest patch I will commit it later in the day.

          Show
          Konstantin Boudnik added a comment - - edited if there are no objections to the latest patch I will commit it later in the day.
          Hide
          Konstantin Boudnik added a comment -

          and it all is fixed now - the bigtop.mk order is preserved during the execution.

          Show
          Konstantin Boudnik added a comment - and it all is fixed now - the bigtop.mk order is preserved during the execution.
          Hide
          Konstantin Boudnik added a comment -

          Found a small issue with the ordering of the dependent targets. Will fix shortly and re-submit the patch. Should be a two line modification compare to the current one.

          Show
          Konstantin Boudnik added a comment - Found a small issue with the ordering of the dependent targets. Will fix shortly and re-submit the patch. Should be a two line modification compare to the current one.
          Hide
          Roman Shaposhnik added a comment -

          Thanks for the answers. LGTM. +1

          Show
          Roman Shaposhnik added a comment - Thanks for the answers. LGTM. +1
          Hide
          Konstantin Boudnik added a comment -

          Adding cool Gradle download-plugin from https://github.com/michel-kraemer/gradle-download-task to get rid of all home-made pesky download logic.

          Show
          Konstantin Boudnik added a comment - Adding cool Gradle download-plugin from https://github.com/michel-kraemer/gradle-download-task to get rid of all home-made pesky download logic.
          Hide
          Konstantin Boudnik added a comment -

          Added the checks to avoid repetition of already performed steps: an idempotent guard of a sort.

          Show
          Konstantin Boudnik added a comment - Added the checks to avoid repetition of already performed steps: an idempotent guard of a sort.
          Hide
          Konstantin Boudnik added a comment -

          the answers in the same order:

          1. fixed
          2. I don't think so, really. I will dig deeper, but I believe this is one of the fixed things in Gradle
          3. I kept current format - and have wrote some of the parsing code - to have both solutions next to each other, so we can gradually phase out make stuff. Once it happens, we surely can change the format of the BOM. In fact, we can totally use Groovy DSL for that, which is a way reacher than JSON or XML; and it will be a way more straight forward to work with. Wadda think?
          4. removed. I still have a comment for a TODO - will fill it in a consequent JIRA is this one is a low priority
          5. Yes, the replacement of Maven has been planned to be done separately. This ticket is for packages only. Do you think it is sensible?
          Show
          Konstantin Boudnik added a comment - the answers in the same order: fixed I don't think so, really. I will dig deeper, but I believe this is one of the fixed things in Gradle I kept current format - and have wrote some of the parsing code - to have both solutions next to each other, so we can gradually phase out make stuff. Once it happens, we surely can change the format of the BOM. In fact, we can totally use Groovy DSL for that, which is a way reacher than JSON or XML; and it will be a way more straight forward to work with. Wadda think? removed. I still have a comment for a TODO - will fill it in a consequent JIRA is this one is a low priority Yes, the replacement of Maven has been planned to be done separately. This ticket is for packages only. Do you think it is sensible?
          Hide
          Roman Shaposhnik added a comment -

          All in all, super useful stuff! In fact, I can't wait to get rid of make logic and get us be 100% java based.

          A few nits/questions:

          1. it seems that AL headers are still missing on a few files in buildSrc/
          2. is buildSrc/ folder at the top level the only way to get this going? IOW, is there any chance to push this stuff to top-level src/ ?
          3. should we migrate bom definition to something more sane, like json? Perhaps not in this patch, but we can totally generate the bigtop.mk from it to keep the make side of things happy for a little while
          4. packages.gradle – should we remove the commented out sections?
          5. at this point, we also have the top level pom.xml driving some of the site generation and project publishing logic. How doable would it be to merge that into gradle?
          Show
          Roman Shaposhnik added a comment - All in all, super useful stuff! In fact, I can't wait to get rid of make logic and get us be 100% java based. A few nits/questions: it seems that AL headers are still missing on a few files in buildSrc/ is buildSrc/ folder at the top level the only way to get this going? IOW, is there any chance to push this stuff to top-level src/ ? should we migrate bom definition to something more sane, like json? Perhaps not in this patch, but we can totally generate the bigtop.mk from it to keep the make side of things happy for a little while packages.gradle – should we remove the commented out sections? at this point, we also have the top level pom.xml driving some of the site generation and project publishing logic. How doable would it be to merge that into gradle?
          Hide
          Konstantin Boudnik added a comment -

          Adding missed ASL2.0

          Show
          Konstantin Boudnik added a comment - Adding missed ASL2.0
          Hide
          Konstantin Boudnik added a comment -

          Yum and Apt repo creation tasks are done, as well as tasks descriptions. The only thing left is support for patching, that can be done separately.

          Please try, review and comment. Thanks!

          Show
          Konstantin Boudnik added a comment - Yum and Apt repo creation tasks are done, as well as tasks descriptions. The only thing left is support for patching, that can be done separately. Please try, review and comment. Thanks!
          Hide
          Konstantin Boudnik added a comment -

          DEB packaging build is fully working now

          Show
          Konstantin Boudnik added a comment - DEB packaging build is fully working now
          Hide
          Konstantin Boudnik added a comment -

          Now SDEB packaging works. One last step....

          Show
          Konstantin Boudnik added a comment - Now SDEB packaging works. One last step....
          Hide
          Konstantin Boudnik added a comment -
          1. I'd say yes - as the deb support is close, let's wait for it
          2. bigtop.mk is the source of the BOM, so we'll have to keep it. As for the make build - I don't mind of keeping it around for some time. Let's perhaps deprecate it?
          Show
          Konstantin Boudnik added a comment - I'd say yes - as the deb support is close, let's wait for it bigtop.mk is the source of the BOM, so we'll have to keep it. As for the make build - I don't mind of keeping it around for some time. Let's perhaps deprecate it?
          Hide
          jay vyas added a comment -

          gotcha. I will play with building it as well.

          1) so we will gate the commit of this on debian ? Or do it in a second patch.
          2) should we delete the "mk" file while commiting this in.

          Show
          jay vyas added a comment - gotcha. I will play with building it as well. 1) so we will gate the commit of this on debian ? Or do it in a second patch. 2) should we delete the "mk" file while commiting this in.
          Hide
          Konstantin Boudnik added a comment -

          Thanks for looking into this.
          README update - yes, definitely needs to be done. I haven't as it is still a work in progress.

          Call-outs to mvn directly - it isn't a part of the package mods, but there should be a better way of doing this - let's address it separately, I guess.

          It would be great to try to build the stack using gradle now, instead of make, so if you can try to play with it - it'd be very nice! I have deb stuff underway now - should be providing an updated patch in a day or so. Please feel free to open new tickets as you're coming across any of the issues with the new build system! Thanks!

          Show
          Konstantin Boudnik added a comment - Thanks for looking into this. README update - yes, definitely needs to be done. I haven't as it is still a work in progress. Call-outs to mvn directly - it isn't a part of the package mods, but there should be a better way of doing this - let's address it separately, I guess. It would be great to try to build the stack using gradle now, instead of make, so if you can try to play with it - it'd be very nice! I have deb stuff underway now - should be providing an updated patch in a day or so. Please feel free to open new tickets as you're coming across any of the issues with the new build system! Thanks!
          Hide
          jay vyas added a comment - - edited

          Hi Cos: Okay finally got to play with this.

          • it works ! I can do "gradle" and it prints all build options. AWESOME !
          • I did a basic test "gradle mahout-download", and it also worked : I saw the mahout tarfile downloaded into the .dl/ directory.

          So 4 initial thoughts :

          • I think now with both "bigtop.mk" "packages.gradle" and "build.gradle" all floating around in the top level, the README definetly needs an update.
          • I see your calling "mvn ..." inside of build.gradle. Is there a non-command line way to invoke maven from gradle? more just a curiosity than anything else. Seems like it would be nice to just require gradle and nothing else, if thats possible.
          • What other things should i do to test this patch? Im just now learning about the bigtop packaging stuff and so don't really know where i should poke at it ....
          • Shall we create a follow up JIRA of next steps which follow this, (i.e. the DEB stuff) as part of pushing this one in?
          Show
          jay vyas added a comment - - edited Hi Cos: Okay finally got to play with this. it works ! I can do "gradle" and it prints all build options. AWESOME ! I did a basic test "gradle mahout-download", and it also worked : I saw the mahout tarfile downloaded into the .dl/ directory. So 4 initial thoughts : I think now with both "bigtop.mk" "packages.gradle" and "build.gradle" all floating around in the top level, the README definetly needs an update. I see your calling "mvn ..." inside of build.gradle. Is there a non-command line way to invoke maven from gradle? more just a curiosity than anything else. Seems like it would be nice to just require gradle and nothing else, if thats possible. What other things should i do to test this patch? Im just now learning about the bigtop packaging stuff and so don't really know where i should poke at it .... Shall we create a follow up JIRA of next steps which follow this, (i.e. the DEB stuff) as part of pushing this one in?
          Hide
          Konstantin Boudnik added a comment - - edited

          It should be pretty straight forward, really. gradle rpms has to provide the same set of packages that make rpms. If there are discrepancies - they will be fixed. Thanks for looking into this!

          Show
          Konstantin Boudnik added a comment - - edited It should be pretty straight forward, really. gradle rpms has to provide the same set of packages that make rpms . If there are discrepancies - they will be fixed. Thanks for looking into this!
          Hide
          jay vyas added a comment -

          i can try to review it , but first have to learn some more about building bigtop, i never did that. maybe during or after sean's packaging class tomorrow i will get a chance to try to build bigtop rpms directly.

          Show
          jay vyas added a comment - i can try to review it , but first have to learn some more about building bigtop, i never did that. maybe during or after sean's packaging class tomorrow i will get a chance to try to build bigtop rpms directly.
          Hide
          Konstantin Boudnik added a comment -

          Guys

          the RPM functionality is ready.
          I still need to implement DEB generation - should be very small change - and patching logic.

          For the sake of bringing this to the community attention it seems to be reasonable to have RPM logic committed and DEB done quickly after as a separate JIRA.

          At any rate - it is ready for review. Please hit it.

          Show
          Konstantin Boudnik added a comment - Guys the RPM functionality is ready. I still need to implement DEB generation - should be very small change - and patching logic. For the sake of bringing this to the community attention it seems to be reasonable to have RPM logic committed and DEB done quickly after as a separate JIRA. At any rate - it is ready for review. Please hit it.
          Hide
          Konstantin Boudnik added a comment -

          Fixed a bug in the download task
          Added all tasks for the same type into a common wrapper. E.g. running
          gradle rpm will run all rpm targets across the stack.
          Fixed a couple of bugs.

          Show
          Konstantin Boudnik added a comment - Fixed a bug in the download task Added all tasks for the same type into a common wrapper. E.g. running gradle rpm will run all rpm targets across the stack. Fixed a couple of bugs.
          Hide
          Konstantin Boudnik added a comment -

          Fixing a couple of TODOs

          Show
          Konstantin Boudnik added a comment - Fixing a couple of TODOs
          Hide
          Konstantin Boudnik added a comment -

          Split up packaging login into its own module, so the main build file isn't so cluttered anymore

          Show
          Konstantin Boudnik added a comment - Split up packaging login into its own module, so the main build file isn't so cluttered anymore
          Hide
          Konstantin Boudnik added a comment -

          Finally, working RPM logic (a couple of TODOs left, but they are sorta minor).

          I will have some improvements done in the next day or so like separation package creation login into a separate file, pretty'fing the build logic, etc. But the overall idea is expressed and can be reviews, criticized, etc. Appreciate the input guys!

          Show
          Konstantin Boudnik added a comment - Finally, working RPM logic (a couple of TODOs left, but they are sorta minor). I will have some improvements done in the next day or so like separation package creation login into a separate file, pretty'fing the build logic, etc. But the overall idea is expressed and can be reviews, criticized, etc. Appreciate the input guys!
          Hide
          Konstantin Boudnik added a comment -

          Another small step closer to the completion

          Show
          Konstantin Boudnik added a comment - Another small step closer to the completion
          Hide
          Konstantin Boudnik added a comment -

          Next iteration - rpms are almost there. The build is shaping up with all fance Gradle calls, etc.

          Show
          Konstantin Boudnik added a comment - Next iteration - rpms are almost there. The build is shaping up with all fance Gradle calls, etc.
          Hide
          Konstantin Boudnik added a comment -

          Initial version of Gradle based build system that (will) replace make in the future. Posting it just to show some progress on the ticket.

          Show
          Konstantin Boudnik added a comment - Initial version of Gradle based build system that (will) replace make in the future. Posting it just to show some progress on the ticket.
          Hide
          Konstantin Boudnik added a comment -

          I am no too naive to try to solve all these aspect in one patch. Hence, this should be considered as an umbrella JIRA.

          In particular, I have a pressing problem whereas a consistent build/installation of the test development and validation environment seems like constantly returning, reproducible issue. Thus, that will be a first sub-task from me.

          Show
          Konstantin Boudnik added a comment - I am no too naive to try to solve all these aspect in one patch. Hence, this should be considered as an umbrella JIRA. In particular, I have a pressing problem whereas a consistent build/installation of the test development and validation environment seems like constantly returning, reproducible issue. Thus, that will be a first sub-task from me.

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development