Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.1.0
    • Component/s: build
    • Labels:
      None

      Description

      Seems confusing to have a .mk file which is mostly just a bunch of variable declarations, which is then parsed as a CSV, simply for the sake of guiding the packages.gradle file .

      Can we be more idiomatic to gradle and either eliminate bigtop.mk by making it into a native gradle data structure (its really just an array, and we can declare in gradle.settings) , so that the readBOM function is easier to follow ?

      I think it is an entry point to understanding bigtop's build system so we should try to simplify it as much as possible to make it maximally easy for people to understand how bigtop's gradle packaging system works.

      1. BIGTOP-1494.patch
        52 kB
        Konstantin Boudnik
      2. BIGTOP-1494.patch
        52 kB
        Konstantin Boudnik
      3. BIGTOP-1494.patch
        51 kB
        Konstantin Boudnik
      4. BIGTOP-1494.patch
        51 kB
        Konstantin Boudnik
      5. BIGTOP-1494.patch
        51 kB
        Konstantin Boudnik
      6. BIGTOP-1494.patch
        50 kB
        Konstantin Boudnik
      7. BIGTOP-1494.patch
        50 kB
        Konstantin Boudnik
      8. BIGTOP-1494.patch
        50 kB
        Konstantin Boudnik
      9. BIGTOP-1494.patch
        25 kB
        Konstantin Boudnik
      10. BIGTOP-1494.patch
        25 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment -

          similar to BIGTOP-1477, possibly can be merged as a minor cleanup task for the packages.gradle file.

          Show
          jayunit100 jay vyas added a comment - similar to BIGTOP-1477 , possibly can be merged as a minor cleanup task for the packages.gradle file.
          Hide
          cos Konstantin Boudnik added a comment -

          Shouldn't be done before we get rid of make

          Show
          cos Konstantin Boudnik added a comment - Shouldn't be done before we get rid of make
          Hide
          cos Konstantin Boudnik added a comment -

          As discussed in BIGTOP-1718 a time for DSL has come. On the same ticket, Rob has implemented the JSON format to represent the stack and component. To refresh it looks like this:

          {
              "version": "0.9.0",
              "components": [
          ...
                  {
                      "name": {
                          "project": "zookeeper",
                          "pkg": "zookeeper",
                          "relNotes": "Apache Zookeeper"
                      },
                      "tarball": {
                          "destination": "zookeeper-3.4.5.tar.gz",
                          "source": "zookeeper-3.4.5.tar.gz"
                      },
                      "url": {
                          "site": "http://apache.osuosl.org/zookeeper/zookeeper-3.4.5",
                          "archive": "http://archive.apache.org/dist/zookeeper/zookeeper-3.4.5"
                      },
                      "version": {
                          "base": "3.4.5",
                          "pkg": "3.4.5",
                          "release": "1"
                      },
                      "git": {
                          "repo": null,
                          "ref": null,
                          "dir": null
                      }
                  }
              ]
          }
          

          I'd say something similar expressed in Groovy should work as a DSL. Thoughts?

          Show
          cos Konstantin Boudnik added a comment - As discussed in BIGTOP-1718 a time for DSL has come. On the same ticket, Rob has implemented the JSON format to represent the stack and component. To refresh it looks like this: { "version" : "0.9.0" , "components" : [ ... { "name" : { "project" : "zookeeper" , "pkg" : "zookeeper" , "relNotes" : "Apache Zookeeper" }, "tarball" : { "destination" : "zookeeper-3.4.5.tar.gz" , "source" : "zookeeper-3.4.5.tar.gz" }, "url" : { "site" : "http: //apache.osuosl.org/zookeeper/zookeeper-3.4.5" , "archive" : "http: //archive.apache.org/dist/zookeeper/zookeeper-3.4.5" }, "version" : { "base" : "3.4.5" , "pkg" : "3.4.5" , "release" : "1" }, "git" : { "repo" : null , "ref" : null , "dir" : null } } ] } I'd say something similar expressed in Groovy should work as a DSL. Thoughts?
          Hide
          jayunit100 jay vyas added a comment -

          Beautiful idea ! Alternatively, for simplicity, a Groovy file with methods combined with an embedded JSON string of BOM components, i.e. BOM.groovy, would work as well. Either way is okay with me !770eba3faf4a

          Show
          jayunit100 jay vyas added a comment - Beautiful idea ! Alternatively, for simplicity, a Groovy file with methods combined with an embedded JSON string of BOM components, i.e. BOM.groovy, would work as well. Either way is okay with me !770eba3faf4a
          Hide
          cos Konstantin Boudnik added a comment -

          Alternatively, for simplicity, a Groovy file with methods combined with an embedded JSON string of BOM components

          that's exactly what Groovy DSL is solving, except that you don't need to write any code

          Show
          cos Konstantin Boudnik added a comment - Alternatively, for simplicity, a Groovy file with methods combined with an embedded JSON string of BOM components that's exactly what Groovy DSL is solving, except that you don't need to write any code
          Hide
          jayunit100 jay vyas added a comment -

          hmm, what do you mean by DSL ? Which DSL ?

          Show
          jayunit100 jay vyas added a comment - hmm, what do you mean by DSL ? Which DSL ?
          Hide
          cos Konstantin Boudnik added a comment -

          I am talking about a Groovy DSL to replace the bigtop.mk completely (which is currently is represented by, essentially, make's DSL). Let me come up with a better thought-through example and I will post here in a bit.

          Show
          cos Konstantin Boudnik added a comment - I am talking about a Groovy DSL to replace the bigtop.mk completely (which is currently is represented by, essentially, make's DSL). Let me come up with a better thought-through example and I will post here in a bit.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          How about something like this

          bigtop {
            stack: [version: "0.9.0, jdk: "1.7.0", scala: "2.10.3", groovy:  "2.4.0"]
            components [
                component {
                  name      project: "bigtop-groovy", pgk; "bigtop-groovy", relNotes; "Grovvy: a dynamic language for the Java platform"
                  tarball   destination: "bigtop-groovy-2.3.8.tar.gz", source: "groovy-binary-2.3.8.zip"
                  url       site: "http://dl.bintray.com/groovy/maven/", archive: "http://dl.bintray.com/groovy/maven/"
                  version   base: "2.3.8", pkg: "2.3.8", release: "1"
                  // Optional, as only null values are specified
                  git       repo: null, ref: null, dir: null
                }
              component {
                ...
              }
              ...
            ]
          }
          

          The code serving this DSL will be pretty straight forward. As an added benefit we can have variety of the stack representations, e.g. JSON, plain text, HTML for the web, etc.

          Thoughts?

          Show
          cos Konstantin Boudnik added a comment - - edited How about something like this bigtop { stack: [version: "0.9.0, jdk: " 1.7.0 ", scala: " 2.10.3 ", groovy: " 2.4.0"] components [ component { name project: "bigtop-groovy" , pgk; "bigtop-groovy" , relNotes; "Grovvy: a dynamic language for the Java platform" tarball destination: "bigtop-groovy-2.3.8.tar.gz" , source: "groovy-binary-2.3.8.zip" url site: "http: //dl.bintray.com/groovy/maven/" , archive: "http://dl.bintray.com/groovy/maven/" version base: "2.3.8" , pkg: "2.3.8" , release: "1" // Optional, as only null values are specified git repo: null , ref: null , dir: null } component { ... } ... ] } The code serving this DSL will be pretty straight forward. As an added benefit we can have variety of the stack representations, e.g. JSON, plain text, HTML for the web, etc. Thoughts?
          Hide
          jayunit100 jay vyas added a comment -

          Looks good to me!

          Show
          jayunit100 jay vyas added a comment - Looks good to me!
          Hide
          kaiyzen Nate DAmico added a comment -

          Is name.project, name.pkg, name.foo, etc a convention with the name prefix for "name of project", "name of package", etc?

          vs

          component {
          name: "bigtop-groovy"
          pgk; "bigtop-groovy",
          relNotes; "Grovvy: a dynamic language for the Java platform"
          }

          Maybe use descr or description vs relNotes (assume that is meant to be related notes?)

          Little off topic, just sticks out looking at the DSL where possibly future evolving around treating the stack items as components themselves. For instance., groovy 2.4.0 is in the stack list, but the component example denotes bigtop-groovy 2.3.8.., would or could something defined in stack be in conflict of something in component?

          Show
          kaiyzen Nate DAmico added a comment - Is name.project, name.pkg, name.foo, etc a convention with the name prefix for "name of project", "name of package", etc? vs component { name: "bigtop-groovy" pgk; "bigtop-groovy", relNotes; "Grovvy: a dynamic language for the Java platform" } Maybe use descr or description vs relNotes (assume that is meant to be related notes?) Little off topic, just sticks out looking at the DSL where possibly future evolving around treating the stack items as components themselves. For instance., groovy 2.4.0 is in the stack list, but the component example denotes bigtop-groovy 2.3.8.., would or could something defined in stack be in conflict of something in component?
          Hide
          cos Konstantin Boudnik added a comment - - edited

          Addressing the point about name.* thing - definitely a miss on my account

          bigtop {
            stack: [version: "0.9.0", jdk: "1.7.0", scala: "2.10.3", groovy:  "2.4.0"]
            components [
                component: [name: "bigtop-groovy", project: "bigtop-groovy", 
                           pgk: "bigtop-groovy", relNotes; "Grovvy: a dynamic language for the Java platform"] {
                  tarball   destination: "bigtop-groovy-2.3.8.tar.gz", source: "groovy-binary-2.3.8.zip"
                  url       site: "http://dl.bintray.com/groovy/maven/", archive: "http://dl.bintray.com/groovy/maven/"
                  version   base: "2.3.8", pkg: "2.3.8", release: "1"
                  // Optional, as only null values are specified
                  git       repo: null, ref: null, dir: null
                }
              component {
                ...
              }
              ...
            ]
          }
          

          the groovy versions in question are for SDK and the runtime to be delivered with the stack. So, in this case it isn't at odds. But potentially it might be the case and that would make to enforce at the object creation level when the DSL is getting consumed? Any other ideas around it?

          Show
          cos Konstantin Boudnik added a comment - - edited Addressing the point about name.* thing - definitely a miss on my account bigtop { stack: [version: "0.9.0" , jdk: "1.7.0" , scala: "2.10.3" , groovy: "2.4.0" ] components [ component: [name: "bigtop-groovy" , project: "bigtop-groovy" , pgk: "bigtop-groovy" , relNotes; "Grovvy: a dynamic language for the Java platform" ] { tarball destination: "bigtop-groovy-2.3.8.tar.gz" , source: "groovy-binary-2.3.8.zip" url site: "http: //dl.bintray.com/groovy/maven/" , archive: "http://dl.bintray.com/groovy/maven/" version base: "2.3.8" , pkg: "2.3.8" , release: "1" // Optional, as only null values are specified git repo: null , ref: null , dir: null } component { ... } ... ] } the groovy versions in question are for SDK and the runtime to be delivered with the stack. So, in this case it isn't at odds. But potentially it might be the case and that would make to enforce at the object creation level when the DSL is getting consumed? Any other ideas around it?
          Hide
          kaiyzen Nate DAmico added a comment -

          ok, makes sense that stack is for the tooling bigtop will use, components what being built.

          with the optional git source reference part, one thing, which would be highly unlikely in public open source setting, but for internal repos orgs might maintain, would be key/auth reference for accessing

          Show
          kaiyzen Nate DAmico added a comment - ok, makes sense that stack is for the tooling bigtop will use, components what being built. with the optional git source reference part, one thing, which would be highly unlikely in public open source setting, but for internal repos orgs might maintain, would be key/auth reference for accessing
          Hide
          cos Konstantin Boudnik added a comment -

          would be highly unlikely in public open source setting

          Yeah. But ppl might be using github, bitbucket, etc.?

          Show
          cos Konstantin Boudnik added a comment - would be highly unlikely in public open source setting Yeah. But ppl might be using github, bitbucket, etc.?
          Hide
          kaiyzen Nate DAmico added a comment -

          Github/bitbucket.., or internal git. For instance, enterprise Foo, is running bigtop internally, and maintain their own src/build of zookeeper b/c they have some tweak or compilation variant they want/need. They could use bigtop but tell it to pull from repo xyz, and either bake the auth/creds in or optionally have bigtop use specific ones denoted. Not sure how relevant or the 0.5% of shops that would want but was thinking through other repo params that might be necessary

          Show
          kaiyzen Nate DAmico added a comment - Github/bitbucket.., or internal git. For instance, enterprise Foo, is running bigtop internally, and maintain their own src/build of zookeeper b/c they have some tweak or compilation variant they want/need. They could use bigtop but tell it to pull from repo xyz, and either bake the auth/creds in or optionally have bigtop use specific ones denoted. Not sure how relevant or the 0.5% of shops that would want but was thinking through other repo params that might be necessary
          Hide
          cos Konstantin Boudnik added a comment -

          True. However, we already have this git option right now (as in 'non-mandatory' option) and I don't see how we can just take it away.

          Show
          cos Konstantin Boudnik added a comment - True. However, we already have this git option right now (as in 'non-mandatory' option) and I don't see how we can just take it away.
          Hide
          cos Konstantin Boudnik added a comment -

          Ok, after looking into this a bit more, here's a bit more polished variance

          bigtop {
            version = "1.1.0"
            stack {
              jdk    = "1.7.0"
              scala  = "2.10.3"
            }
            components {
              component {
                name    = "bigtop-groovy"
                project = name
                version {
                  base = "2.4.4"
                  pkg = base
                  release = "1"
                }
                pgk = "bigtop-groovy"
                relNotes = "Groovy: a dynamic language for the Java platform"
                tarball {
                  destination = "bigtop-groovy-${version.base}.tar.gz"
                  source = "apache-groovy-binary-${version.base}.zip"
                }
                url {
                  site = "http://dl.bintray.com/groovy/maven/"
                  archive = site
                }
                // Optional, as only null values are specified
                git {
                  repo = null
                  ref = null
                  dir = null
                }
              }
              component {
                name    = "bigtop-utils"
                project = name
                version {
                  base = bigtop.version
                  pkg = base
                  release = "1"
                }
                pgk = "bigtop-utils"
                relNotes = "Service package for Apache Bigtop runtime"
                tarball {
                  destination = "bigtop-utils-${version.base}.tar.gz"
                }
              }
            }
          }
          
          Show
          cos Konstantin Boudnik added a comment - Ok, after looking into this a bit more, here's a bit more polished variance bigtop { version = "1.1.0" stack { jdk = "1.7.0" scala = "2.10.3" } components { component { name = "bigtop-groovy" project = name version { base = "2.4.4" pkg = base release = "1" } pgk = "bigtop-groovy" relNotes = "Groovy: a dynamic language for the Java platform" tarball { destination = "bigtop-groovy-${version.base}.tar.gz" source = "apache-groovy-binary-${version.base}.zip" } url { site = "http: //dl.bintray.com/groovy/maven/" archive = site } // Optional, as only null values are specified git { repo = null ref = null dir = null } } component { name = "bigtop-utils" project = name version { base = bigtop.version pkg = base release = "1" } pgk = "bigtop-utils" relNotes = "Service package for Apache Bigtop runtime" tarball { destination = "bigtop-utils-${version.base}.tar.gz" } } } }
          Hide
          jayunit100 jay vyas added a comment -

          it's a good start and I think if it's working we can move with it.

          Olaf should chime in here.... I know he builds bigtop from beginning to end and probably uses the malefile.

          Others?

          Show
          jayunit100 jay vyas added a comment - it's a good start and I think if it's working we can move with it. Olaf should chime in here.... I know he builds bigtop from beginning to end and probably uses the malefile. Others?
          Hide
          cos Konstantin Boudnik added a comment -

          This format works with standard ConfigSlurper, so we don't need write much of the code to parse and use it.

          Show
          cos Konstantin Boudnik added a comment - This format works with standard ConfigSlurper, so we don't need write much of the code to parse and use it.
          Hide
          cos Konstantin Boudnik added a comment -

          Here's the even more polished variant of the new BOM configuration format.

          Unable to find source-code formatter for language: groovy. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
          bigtop {
          
          /** Base Configuration of the mirror and archives */
              final def APACHE_MIRROR  = "http://apache.osuosl.org"
              final def APACHE_ARCHIVE = "http://archive.apache.org/dist"
          /** End of Base Configuration */
          
            version = "1.1.0"
            stack   = [jdk:"1.7.0", scala:"2.10.3"]
            components {
              'bigtop-groovy' {
                name    = 'bigtop-groovy'
                pgk     = 'bigtop-groovy'
                version = [base: '2.4.4', pkg: '2.4.4', release: 1]
                relNotes = "Groovy: a dynamic language for the Java platform"
                tarball = [destination: "bigtop-groovy-${version.base}.tar.gz",
                           source: "apache-groovy-binary-${version.base}.zip"]
                url     = [site: "http://dl.bintray.com/groovy/maven/", archive: site]
                // Optional, as only null values are specified
                git     = [repo: null,ref: null, dir: null]
              }
              'bigtop-utils' {
                name    = "bigtop-utils"
                pgk     = "bigtop-utils"
                version { base = bigtop.version; pkg = base; release = 1 }
                relNotes = "Service package for Apache Bigtop runtime"
                tarball { destination = "bigtop-utils-${version.base}.tar.gz" }
              }
              'zookeeper' {
                name    = 'zookeeper'
                pgk     = name
                version {
                  base  = '3.4.6'
                  pkg   = base
                  release = 1
                }
                tarball {
                  source      = "zookeeper-${version.base}.tar.gz"
                  destination = source
                }
                url {
                  download_path = "/zookeeper/zookeeper-${version.base}"
                  site          = "${APACHE_MIRROR}/${download_path}"
                  archive       = "${APACHE_ARCHIVE}/${download_path}"
                }
              }
            }
          }
          
          Show
          cos Konstantin Boudnik added a comment - Here's the even more polished variant of the new BOM configuration format. Unable to find source-code formatter for language: groovy. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml bigtop { /** Base Configuration of the mirror and archives */ final def APACHE_MIRROR = "http: //apache.osuosl.org" final def APACHE_ARCHIVE = "http: //archive.apache.org/dist" /** End of Base Configuration */ version = "1.1.0" stack = [jdk: "1.7.0" , scala: "2.10.3" ] components { 'bigtop-groovy' { name = 'bigtop-groovy' pgk = 'bigtop-groovy' version = [base: '2.4.4', pkg: '2.4.4', release: 1] relNotes = "Groovy: a dynamic language for the Java platform" tarball = [destination: "bigtop-groovy-${version.base}.tar.gz" , source: "apache-groovy-binary-${version.base}.zip" ] url = [site: "http: //dl.bintray.com/groovy/maven/" , archive: site] // Optional, as only null values are specified git = [repo: null ,ref: null , dir: null ] } 'bigtop-utils' { name = "bigtop-utils" pgk = "bigtop-utils" version { base = bigtop.version; pkg = base; release = 1 } relNotes = "Service package for Apache Bigtop runtime" tarball { destination = "bigtop-utils-${version.base}.tar.gz" } } 'zookeeper' { name = 'zookeeper' pgk = name version { base = '3.4.6' pkg = base release = 1 } tarball { source = "zookeeper-${version.base}.tar.gz" destination = source } url { download_path = "/zookeeper/zookeeper-${version.base}" site = "${APACHE_MIRROR}/${download_path}" archive = "${APACHE_ARCHIVE}/${download_path}" } } } }
          Hide
          cos Konstantin Boudnik added a comment - - edited

          Here's the even more polished variant of the new BOM configuration format.

          bigtop {
          
          /** Base Configuration of the mirror and archives */
              final def APACHE_MIRROR  = "http://apache.osuosl.org"
              final def APACHE_ARCHIVE = "http://archive.apache.org/dist"
          /** End of Base Configuration */
          
            version = "1.1.0"
            stack   = [jdk:"1.7.0", scala:"2.10.3"]
            components {
              'bigtop-groovy' {
                name    = 'bigtop-groovy'
                pgk     = 'bigtop-groovy'
                version = [base: '2.4.4', pkg: '2.4.4', release: 1]
                relNotes = "Groovy: a dynamic language for the Java platform"
                tarball = [destination: "bigtop-groovy-${version.base}.tar.gz",
                           source: "apache-groovy-binary-${version.base}.zip"]
                url     = [site: "http://dl.bintray.com/groovy/maven/", archive: site]
                // Optional, as only null values are specified
                git     = [repo: null,ref: null, dir: null]
              }
              'bigtop-utils' {
                name    = "bigtop-utils"
                pgk     = "bigtop-utils"
                version { base = bigtop.version; pkg = base; release = 1 }
                relNotes = "Service package for Apache Bigtop runtime"
                tarball { destination = "bigtop-utils-${version.base}.tar.gz" }
              }
              'zookeeper' {
                name    = 'zookeeper'
                pgk     = name
                version {
                  base  = '3.4.6'
                  pkg   = base
                  release = 1
                }
                tarball {
                  source      = "zookeeper-${version.base}.tar.gz"
                  destination = source
                }
                url {
                  download_path = "/zookeeper/zookeeper-${version.base}"
                  site          = "${APACHE_MIRROR}/${download_path}"
                  archive       = "${APACHE_ARCHIVE}/${download_path}"
                }
              }
            }
          }
          

          The the code to access it is getting very simple. Here's a short example

          class config {
            def static BOM = new ConfigSlurper().parse(new URL('file:///home/cos/workspaces/dsl-test/bigtop.bom'))
          
            public static void main(String ... args) {
              println BOM.bigtop.stack
              println BOM.bigtop.stack.jdk
              BOM.bigtop.components.each { name, comp ->
                println "$name ${comp.version} ${comp.tarball.destination}"
                println comp.url.site
              }
            }
          }
          

          I am going to proceed with the initial implementation to replace bigtop.mk and most of the packages.gradle logic

          Show
          cos Konstantin Boudnik added a comment - - edited Here's the even more polished variant of the new BOM configuration format. bigtop { /** Base Configuration of the mirror and archives */ final def APACHE_MIRROR = "http: //apache.osuosl.org" final def APACHE_ARCHIVE = "http: //archive.apache.org/dist" /** End of Base Configuration */ version = "1.1.0" stack = [jdk: "1.7.0" , scala: "2.10.3" ] components { 'bigtop-groovy' { name = 'bigtop-groovy' pgk = 'bigtop-groovy' version = [base: '2.4.4', pkg: '2.4.4', release: 1] relNotes = "Groovy: a dynamic language for the Java platform" tarball = [destination: "bigtop-groovy-${version.base}.tar.gz" , source: "apache-groovy-binary-${version.base}.zip" ] url = [site: "http: //dl.bintray.com/groovy/maven/" , archive: site] // Optional, as only null values are specified git = [repo: null ,ref: null , dir: null ] } 'bigtop-utils' { name = "bigtop-utils" pgk = "bigtop-utils" version { base = bigtop.version; pkg = base; release = 1 } relNotes = "Service package for Apache Bigtop runtime" tarball { destination = "bigtop-utils-${version.base}.tar.gz" } } 'zookeeper' { name = 'zookeeper' pgk = name version { base = '3.4.6' pkg = base release = 1 } tarball { source = "zookeeper-${version.base}.tar.gz" destination = source } url { download_path = "/zookeeper/zookeeper-${version.base}" site = "${APACHE_MIRROR}/${download_path}" archive = "${APACHE_ARCHIVE}/${download_path}" } } } } The the code to access it is getting very simple. Here's a short example class config { def static BOM = new ConfigSlurper().parse( new URL('file: ///home/cos/workspaces/dsl-test/bigtop.bom')) public static void main( String ... args) { println BOM.bigtop.stack println BOM.bigtop.stack.jdk BOM.bigtop.components.each { name, comp -> println "$name ${comp.version} ${comp.tarball.destination}" println comp.url.site } } } I am going to proceed with the initial implementation to replace bigtop.mk and most of the packages.gradle logic
          Hide
          cos Konstantin Boudnik added a comment -

          The latest examples represents three different styles of the DSL, but the processing code is the same. The one presented in "bigtop-groovy" seems to be the most compact, but I personally enjoy 2nd and 3rd, because it allows me to refer to already defined properties like here

                version {
                  base  = '3.4.6'
                  pkg   = base
                  release = 1
                }
          
          Show
          cos Konstantin Boudnik added a comment - The latest examples represents three different styles of the DSL, but the processing code is the same. The one presented in "bigtop-groovy" seems to be the most compact, but I personally enjoy 2nd and 3rd, because it allows me to refer to already defined properties like here version { base = '3.4.6' pkg = base release = 1 }
          Hide
          cos Konstantin Boudnik added a comment -

          Initial patch. All functionality works as before using new configuration format represented by bigtop.bom file. I still need to populate it with all the components in the stack. Perhaps need to add some checks for the sanity of the incoming bom file.

          Current version is transitory and is made to avoid any major changes in the underlying components builds. That's why I am flattening the configuration structure to a map and using all these nasty FULLY_CAPITALIZED_WITH_UNDERBARS names for the varialbles.

          Would appreciate any early feedback. Thanks!

          Show
          cos Konstantin Boudnik added a comment - Initial patch. All functionality works as before using new configuration format represented by bigtop.bom file. I still need to populate it with all the components in the stack. Perhaps need to add some checks for the sanity of the incoming bom file. Current version is transitory and is made to avoid any major changes in the underlying components builds. That's why I am flattening the configuration structure to a map and using all these nasty FULLY_CAPITALIZED_WITH_UNDERBARS names for the varialbles. Would appreciate any early feedback. Thanks!
          Hide
          cos Konstantin Boudnik added a comment -

          A bit improved version

          Show
          cos Konstantin Boudnik added a comment - A bit improved version
          Hide
          cos Konstantin Boudnik added a comment -

          The BOM is fully based on new configuration now and old mk file is removed.

          Show
          cos Konstantin Boudnik added a comment - The BOM is fully based on new configuration now and old mk file is removed.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          A typo on jsvc download path

          Show
          cos Konstantin Boudnik added a comment - - edited A typo on jsvc download path
          Hide
          cos Konstantin Boudnik added a comment -

          Let's review it.

          Show
          cos Konstantin Boudnik added a comment - Let's review it.
          Hide
          cos Konstantin Boudnik added a comment -

          I have added some DSL documentation and minimal BOM validations. Now it is ready IMO

          Show
          cos Konstantin Boudnik added a comment - I have added some DSL documentation and minimal BOM validations. Now it is ready IMO
          Hide
          rvs Roman Shaposhnik added a comment -

          This looks pretty awesome to me! Thanks for getting it done.

          Like you said – there are a few improvements I can see on top of the new format (like auto-generating default parts of configuration, etc.) but I'd like to transition first and do improvements later.

          Show
          rvs Roman Shaposhnik added a comment - This looks pretty awesome to me! Thanks for getting it done. Like you said – there are a few improvements I can see on top of the new format (like auto-generating default parts of configuration, etc.) but I'd like to transition first and do improvements later.
          Hide
          cos Konstantin Boudnik added a comment -

          I take it as +1 Thanks for the review and I agree with the sentiment. That's why I moved this into a sub-task, so we can gradually improve on the initial implementation.

          Before committing, I still need to run a full build for a deb and rpm parts to make sure that new definitions aren't missing anything. That's where the damn CI will fit perfectly. Oh well...

          Show
          cos Konstantin Boudnik added a comment - I take it as +1 Thanks for the review and I agree with the sentiment. That's why I moved this into a sub-task, so we can gradually improve on the initial implementation. Before committing, I still need to run a full build for a deb and rpm parts to make sure that new definitions aren't missing anything. That's where the damn CI will fit perfectly. Oh well...
          Hide
          cos Konstantin Boudnik added a comment -

          Ironed out a couple of wrinkles in the tarball names and such. deb is building fully now followed by apt.

          I don't have RPM-based environment ready, so if anyone can do ./gradlew rpm yum to make sure the build goes trough - we can get this in. Otherwise, I will try to validate RPMs tomorrow. Thanks!

          Show
          cos Konstantin Boudnik added a comment - Ironed out a couple of wrinkles in the tarball names and such. deb is building fully now followed by apt. I don't have RPM-based environment ready, so if anyone can do ./gradlew rpm yum to make sure the build goes trough - we can get this in. Otherwise, I will try to validate RPMs tomorrow. Thanks!
          Hide
          rnowling RJ Nowling added a comment - - edited

          I'm starting a build. I'll let you know how it goes.

          Update: I'm getting an error when trying to apply the patch:

          fatal: git diff header lacks filename information when removing 1 leading pathname component (line 6)

          Show
          rnowling RJ Nowling added a comment - - edited I'm starting a build. I'll let you know how it goes. Update: I'm getting an error when trying to apply the patch: fatal: git diff header lacks filename information when removing 1 leading pathname component (line 6)
          Hide
          cos Konstantin Boudnik added a comment -

          It isn't git-formatted just yet - please just use
          % patch -p0 < BIGTOP-1494.patch

          or
          % git apply -p0 < BIGTOP-1494.patch

          Show
          cos Konstantin Boudnik added a comment - It isn't git-formatted just yet - please just use % patch -p0 < BIGTOP-1494 .patch or % git apply -p0 < BIGTOP-1494 .patch
          Hide
          rnowling RJ Nowling added a comment -

          And off we go!

          Show
          rnowling RJ Nowling added a comment - And off we go!
          Hide
          rnowling RJ Nowling added a comment - - edited

          So far, I'm running into the following issues:

          1) For tomcat, $name is set to bigtop-tomcat but tomcat is expected in the source, destination, and download paths.

          2) For tomcat, -native needs to be removed from the source

          3) RPMs report that versions with hyphens ( - ) are not allowed. Had to remove -SNAPSHOT from the Bigtop version.

          Show
          rnowling RJ Nowling added a comment - - edited So far, I'm running into the following issues: 1) For tomcat, $name is set to bigtop-tomcat but tomcat is expected in the source, destination, and download paths. 2) For tomcat, -native needs to be removed from the source 3) RPMs report that versions with hyphens ( - ) are not allowed. Had to remove -SNAPSHOT from the Bigtop version.
          Hide
          cos Konstantin Boudnik added a comment -

          Fixed #1 and #2 (my bad - fixed and missed the test on this)
          for #3 the following

          -      version { base = bigtop.version; pkg = base; release = 1 }
          +      version { base = bigtop.version; pkg = base-"-SNAPSHOT"; release = 1 }
          

          does the trick. Oh mighty Groovy! Validated and it works.

          Thanks for the validation man! I will commit this tomorrow am unless here any objections here.

          Show
          cos Konstantin Boudnik added a comment - Fixed #1 and #2 (my bad - fixed and missed the test on this) for #3 the following - version { base = bigtop.version; pkg = base; release = 1 } + version { base = bigtop.version; pkg = base- "-SNAPSHOT" ; release = 1 } does the trick. Oh mighty Groovy! Validated and it works. Thanks for the validation man! I will commit this tomorrow am unless here any objections here.
          Hide
          rnowling RJ Nowling added a comment -

          Sorry, Cos but not done yet! I've run into an issue with datafu. I think the pkg variable needs to be set to pig-udf-datafu. Trying that now and rerunning the build.

          Show
          rnowling RJ Nowling added a comment - Sorry, Cos but not done yet! I've run into an issue with datafu. I think the pkg variable needs to be set to pig-udf-datafu . Trying that now and rerunning the build.
          Hide
          cos Konstantin Boudnik added a comment -

          That's weird as it worked for deb... But it could be something of those differences. Thanks!

          Show
          cos Konstantin Boudnik added a comment - That's weird as it worked for deb... But it could be something of those differences. Thanks!
          Hide
          rnowling RJ Nowling added a comment -

          Ok, I can confirm – setting the pkg variable fixed the issue with datafu.

          Giraph seemed to get into an infinite loop. Maven kept reporting "something changed, rebuilding!" and the same jars would be built over and over. I'm not sure if this is related to the changes in this patch, though.

          Other than Giraph, I built most of the RPMs successfully and feel that at least the major bugs have been worked out. We'll just need to keep an eye on build issues for a while to make sure we get everything before the next release.

          +1 to commit once datafu stuff is fixed.

          Show
          rnowling RJ Nowling added a comment - Ok, I can confirm – setting the pkg variable fixed the issue with datafu. Giraph seemed to get into an infinite loop. Maven kept reporting "something changed, rebuilding!" and the same jars would be built over and over. I'm not sure if this is related to the changes in this patch, though. Other than Giraph, I built most of the RPMs successfully and feel that at least the major bugs have been worked out. We'll just need to keep an eye on build issues for a while to make sure we get everything before the next release. +1 to commit once datafu stuff is fixed.
          Hide
          cos Konstantin Boudnik added a comment -

          Fixed the dataFu. Also, ran "giraph-rpm" and it seems to be doing all right: it seems to be building an awful lot of jars for hadoop, hbase, hcatalog, hive, and other things and takes a while. I am hitting the following:

          Could not transfer artifact org.hyperic:sigar:pom:1.6.5.132-5
          

          but it seems like my local networking issue or something and isn't related to the patch by any stretch.

          New patch is getting attached. I will set it sit until tomorrow in case folks in other timezones want to test and find some issues

          Show
          cos Konstantin Boudnik added a comment - Fixed the dataFu. Also, ran "giraph-rpm" and it seems to be doing all right: it seems to be building an awful lot of jars for hadoop, hbase, hcatalog, hive, and other things and takes a while. I am hitting the following: Could not transfer artifact org.hyperic:sigar:pom:1.6.5.132-5 but it seems like my local networking issue or something and isn't related to the patch by any stretch. New patch is getting attached. I will set it sit until tomorrow in case folks in other timezones want to test and find some issues
          Hide
          cos Konstantin Boudnik added a comment -

          Final (hopefully) version of the patch, properly formatted, etc.

          Show
          cos Konstantin Boudnik added a comment - Final (hopefully) version of the patch, properly formatted, etc.
          Hide
          cos Konstantin Boudnik added a comment -

          Last version of the patch which is rebased with the consideration for Hadoop's upgrade to 2.7.1 from db3e38a

          Show
          cos Konstantin Boudnik added a comment - Last version of the patch which is rebased with the consideration for Hadoop's upgrade to 2.7.1 from db3e38a
          Hide
          cos Konstantin Boudnik added a comment -

          Pushed to the master. Thanks for the review and help with testing.

          Show
          cos Konstantin Boudnik added a comment - Pushed to the master. Thanks for the review and help with testing.
          Hide
          rnowling RJ Nowling added a comment -

          No problem!

          Show
          rnowling RJ Nowling added a comment - No problem!

            People

            • Assignee:
              cos Konstantin Boudnik
              Reporter:
              jayunit100 jay vyas
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development