Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: build
    • Labels:
      None

      Description

      It might be a good idea to be able to set up a dependencies between the components, so the build system is aware which parts should be build first, and the sequentially or in parallel.

      With new configuration DSL introduced in BIGTOP-1494 we can have a directional graph like this

          'hadoop' {
            name    = 'hadoop'
            relNotes = 'Apache Hadoop'
            version { base = '2.6.0'; pkg = base; release =1 }
            tarball { destination = "${name}-${version.base}.tar.gz"
                      source      = "${name}-${version.base}-src.tar.gz" }
            url     { site = "{apache.APACHE_MIRROR}/${download_path}"
                      archive = "{apache.APACHE_ARCHIVE}/${download_path}"
                      download_path = "/hadoop/common/$name-${version.base}" }
            dependencies { 'zookeeper'}
          }
      

      which will allow to collect all dependencies information cross components, build the graph and test it for loops. This information can be used by the build system to order the build process: if hadoop-deb is invoked it will automatically tries to build zookeeper; for hbase it will invoke hadoop build first, etc.

      1. BIGTOP-2025.patch
        5 kB
        Konstantin Boudnik
      2. BIGTOP-2025.patch
        5 kB
        Konstantin Boudnik
      3. BIGTOP-2025.patch
        4 kB
        Konstantin Boudnik
      4. BIGTOP-2025.patch
        4 kB
        Konstantin Boudnik
      5. BIGTOP-2025.patch
        4 kB
        Konstantin Boudnik
      6. BIGTOP-2025.patch
        5 kB
        Konstantin Boudnik
      7. BIGTOP-2025.patch
        3 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          Pushed to the master. Thanks for everybody's help

          Show
          cos Konstantin Boudnik added a comment - Pushed to the master. Thanks for everybody's help
          Hide
          cos Konstantin Boudnik added a comment -

          I am going to commit BIGTOP-2051 in a couple of hours, followed by this one. In case there are any other comments...

          Show
          cos Konstantin Boudnik added a comment - I am going to commit BIGTOP-2051 in a couple of hours, followed by this one. In case there are any other comments...
          Hide
          oflebbe Olaf Flebbe added a comment - - edited

          I think

          gradle clean rpm

          Will force a full rebuild.

          Show
          oflebbe Olaf Flebbe added a comment - - edited I think gradle clean rpm Will force a full rebuild.
          Hide
          cos Konstantin Boudnik added a comment -

          Updating the README.md as well.

          Show
          cos Konstantin Boudnik added a comment - Updating the README.md as well.
          Hide
          cos Konstantin Boudnik added a comment -

          Ok, the latest patch adds a way to avoid building a component dependencies with

          -Dbuildnodeps=true
          

          By default the order set by dependencies will be followed. The rest of the functionality as review before - apologies for the changes.

          Show
          cos Konstantin Boudnik added a comment - Ok, the latest patch adds a way to avoid building a component dependencies with -Dbuildnodeps= true By default the order set by dependencies will be followed. The rest of the functionality as review before - apologies for the changes.
          Hide
          kaiyzen Nate DAmico added a comment -

          One thing to consider along with a no dependencies flag is once we have artifacts cached across builds, probably would need a "build all" flag so it would ignore the caches and force rebuild on the dependents.

          Show
          kaiyzen Nate DAmico added a comment - One thing to consider along with a no dependencies flag is once we have artifacts cached across builds, probably would need a "build all" flag so it would ignore the caches and force rebuild on the dependents.
          Hide
          cos Konstantin Boudnik added a comment -

          I am testing phoenix-rpm build to validate BIGTOP-2051 and running gradle phoenix-rpm triggers zookeeper, hadoop, and hbase builds. Pretty neat. May be it would make sense to add a switch to turnoff the directed build? Something like gradle -Dnodeps or similar? Thoughts?

          Show
          cos Konstantin Boudnik added a comment - I am testing phoenix-rpm build to validate BIGTOP-2051 and running gradle phoenix-rpm triggers zookeeper, hadoop, and hbase builds. Pretty neat. May be it would make sense to add a switch to turnoff the directed build? Something like gradle -Dnodeps or similar? Thoughts?
          Hide
          apurtell Andrew Purtell added a comment -

          Not ideal, but that's not bad for a start.

          lgtm

          Show
          apurtell Andrew Purtell added a comment - Not ideal, but that's not bad for a start. lgtm
          Hide
          cos Konstantin Boudnik added a comment -

          What's gonna happen if there's a circular deps, the build will fail with something like

          * What went wrong:
          Execution failed for task ':tasks'.
          > java.lang.StackOverflowError (no error message)
          

          Not ideal, and perhaps should be addressed. May be separately?

          Show
          cos Konstantin Boudnik added a comment - What's gonna happen if there's a circular deps, the build will fail with something like * What went wrong: Execution failed for task ':tasks'. > java.lang.StackOverflowError (no error message) Not ideal, and perhaps should be addressed. May be separately?
          Hide
          apurtell Andrew Purtell added a comment -

          Does this detect accidental cycles? I don't see it, but I'm not super familiar with gradle.

          Show
          apurtell Andrew Purtell added a comment - Does this detect accidental cycles? I don't see it, but I'm not super familiar with gradle.
          Hide
          cos Konstantin Boudnik added a comment -

          Documenting the syntax

          Show
          cos Konstantin Boudnik added a comment - Documenting the syntax
          Hide
          cos Konstantin Boudnik added a comment -

          I will add a comment to the dependency section of the bom file itself before committing. Thanks!

          Show
          cos Konstantin Boudnik added a comment - I will add a comment to the dependency section of the bom file itself before committing. Thanks!
          Hide
          oflebbe Olaf Flebbe added a comment -

          Its too late now ... I am confused .

          Perhaps a comment how to read the dependency order in the code would help to clarify this a bit.

          You are perfectly right and the +1 still stands.

          Show
          oflebbe Olaf Flebbe added a comment - Its too late now ... I am confused . Perhaps a comment how to read the dependency order in the code would help to clarify this a bit. You are perfectly right and the +1 still stands.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          The way you express it is opposite to what you're asked for. The syntax is
          dependsOn:=[dependents]
          Your version will make hive to be depending on pig, not the other way around

          Show
          cos Konstantin Boudnik added a comment - - edited The way you express it is opposite to what you're asked for. The syntax is dependsOn:= [dependents] Your version will make hive to be depending on pig, not the other way around
          Hide
          oflebbe Olaf Flebbe added a comment -

          We may fine tune the dependencies later.

          +1 from me, thank you for your patience.

          Show
          oflebbe Olaf Flebbe added a comment - We may fine tune the dependencies later. +1 from me, thank you for your patience.
          Hide
          oflebbe Olaf Flebbe added a comment - - edited

          Sorry that I wasn't clear enough.

          oozie: ['hive']
          pig: ['hive']
          

          We may drop the oozie dependency since oozie depends on an ancient hive version.

          Show
          oflebbe Olaf Flebbe added a comment - - edited Sorry that I wasn't clear enough. oozie: ['hive'] pig: ['hive'] We may drop the oozie dependency since oozie depends on an ancient hive version.
          Hide
          cos Konstantin Boudnik added a comment -

          Ah, makes sense - thanks!. So, I presume this one is ok, once BIGTOP-2051 is in?

          Show
          cos Konstantin Boudnik added a comment - Ah, makes sense - thanks!. So, I presume this one is ok, once BIGTOP-2051 is in?
          Hide
          oflebbe Olaf Flebbe added a comment -

          hue is a python application. It uses the Web UI and REST interfaces and does not consume java code.

          In fact it contains java code, but that is the obsolete MR1 compatible cruft we do not use at all.

          Show
          oflebbe Olaf Flebbe added a comment - hue is a python application. It uses the Web UI and REST interfaces and does not consume java code. In fact it contains java code, but that is the obsolete MR1 compatible cruft we do not use at all.
          Hide
          cos Konstantin Boudnik added a comment -

          Fixed dependencies for pig and hue. Why hue doesn't depend on Hadoop is beyond me, but I have any idea about that piece anyway, so just following your lead. Thanks

          Show
          cos Konstantin Boudnik added a comment - Fixed dependencies for pig and hue. Why hue doesn't depend on Hadoop is beyond me, but I have any idea about that piece anyway, so just following your lead. Thanks
          Hide
          oflebbe Olaf Flebbe added a comment - - edited

          Yes pig depends on hive. I had to open a JIRA because of that.

          PIG-4629

          hue should not depend on hadoop

          Show
          oflebbe Olaf Flebbe added a comment - - edited Yes pig depends on hive. I had to open a JIRA because of that. PIG-4629 hue should not depend on hadoop
          Hide
          cos Konstantin Boudnik added a comment -

          I take it back:

          • hue already has hadoop dependency
          • is hive really required in order to build pig? I am not aware of this, but can update the dependency, of course
          Show
          cos Konstantin Boudnik added a comment - I take it back: hue already has hadoop dependency is hive really required in order to build pig? I am not aware of this, but can update the dependency, of course
          Hide
          oflebbe Olaf Flebbe added a comment -

          I am sure my machine will explode...

          Show
          oflebbe Olaf Flebbe added a comment - I am sure my machine will explode...
          Hide
          cos Konstantin Boudnik added a comment -

          Running it with "./gradle --parallel" wasn't set as a requirement for this, really. I am not even sure if that feature works in gradle yet.
          Will update the dependencies shortly.

          Show
          cos Konstantin Boudnik added a comment - Running it with "./gradle --parallel" wasn't set as a requirement for this, really. I am not even sure if that feature works in gradle yet. Will update the dependencies shortly.
          Hide
          oflebbe Olaf Flebbe added a comment - - edited
          • Hue has no hadoop dependency
          • pig depends on hive

          Patch does not apply cleanly

          Does not work as expected
          gradle --parallel deb

          Show
          oflebbe Olaf Flebbe added a comment - - edited Hue has no hadoop dependency pig depends on hive Patch does not apply cleanly Does not work as expected gradle --parallel deb
          Hide
          cos Konstantin Boudnik added a comment -

          The latest implementation sits on top of BIGTOP-2025. Testing is easy by running
          ./gradlew tasks --all to see the dependencies getting aligned

          Show
          cos Konstantin Boudnik added a comment - The latest implementation sits on top of BIGTOP-2025 . Testing is easy by running ./gradlew tasks --all to see the dependencies getting aligned
          Hide
          jayunit100 jay vyas added a comment -

          +1 but untested .

          this is a cool feature.

          Show
          jayunit100 jay vyas added a comment - +1 but untested . this is a cool feature.
          Hide
          cos Konstantin Boudnik added a comment -

          Now with fall-back behavior

          Show
          cos Konstantin Boudnik added a comment - Now with fall-back behavior
          Hide
          kaiyzen Nate DAmico added a comment -

          Makes sense to have some fallback default. Always good UX to do best effort when users omit input

          Show
          kaiyzen Nate DAmico added a comment - Makes sense to have some fallback default. Always good UX to do best effort when users omit input
          Hide
          cos Konstantin Boudnik added a comment -

          Initial version of the dependencies direction.
          I am debating with myself if having an ability to fall back to the old way of ordering the build by the position of a component in the BOM would makes sense. Perhaps for situations when dependencies aren't specified. Thoughts?

          Show
          cos Konstantin Boudnik added a comment - Initial version of the dependencies direction. I am debating with myself if having an ability to fall back to the old way of ordering the build by the position of a component in the BOM would makes sense. Perhaps for situations when dependencies aren't specified. Thoughts?
          Hide
          cos Konstantin Boudnik added a comment -

          Thanks Nate! Here's the link to that particular comment and consequent conversation ....

          Show
          cos Konstantin Boudnik added a comment - Thanks Nate! Here's the link to that particular comment and consequent conversation ....
          Hide
          kaiyzen Nate DAmico added a comment -
          Show
          kaiyzen Nate DAmico added a comment - Some convo was on this jira: https://issues.apache.org/jira/browse/BIGTOP-1906
          Hide
          cos Konstantin Boudnik added a comment -

          Yup, I wish I remember where that conversation was carries over initially, so we can link it here. If anyone has the reference to that conversation - please share!

          Show
          cos Konstantin Boudnik added a comment - Yup, I wish I remember where that conversation was carries over initially, so we can link it here. If anyone has the reference to that conversation - please share!
          Hide
          kaiyzen Nate DAmico added a comment -

          Seems like this plus components publishing to local maven repo would provide everything needs for more streamlined builds that aren't doing the kitchen sink

          Show
          kaiyzen Nate DAmico added a comment - Seems like this plus components publishing to local maven repo would provide everything needs for more streamlined builds that aren't doing the kitchen sink
          Hide
          warwithin YoungWoo Kim added a comment -

          That's a great idea!

          Show
          warwithin YoungWoo Kim added a comment - That's a great idea!

            People

            • Assignee:
              cos Konstantin Boudnik
              Reporter:
              cos Konstantin Boudnik
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development