Bigtop
  1. Bigtop
  2. BIGTOP-98

Ability to force ivy/maven version inter-dependency needs to be implemented

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We need to have a mechanism in place to be able to force compile-time inter-dependencies to be within stack of Bigtop components. Currently we rely on whatever projects specify at release time. This leads to an unfortunate side effect of components compiling against one version of artifacts and deploying against a different one.

        Activity

        Hide
        Patrick Hunt added a comment -

        No, but it is attached to the the ZOOKEEPER-1078 jira. I'm planning to commit it next I have some free time.

        Show
        Patrick Hunt added a comment - No, but it is attached to the the ZOOKEEPER-1078 jira. I'm planning to commit it next I have some free time.
        Hide
        Andrew Bayer added a comment -

        Any chance that ZK patch is in a Github branch somewhere for me to play with?

        Show
        Andrew Bayer added a comment - Any chance that ZK patch is in a Github branch somewhere for me to play with?
        Hide
        Patrick Hunt added a comment -

        FYI ZK and Maven, patch pending: https://issues.apache.org/jira/browse/ZOOKEEPER-1078

        I'd love it if someone were to jump in and help out with this effort. (it's on trunk so it would be for the ZK 3.5.0 release)

        Show
        Patrick Hunt added a comment - FYI ZK and Maven, patch pending: https://issues.apache.org/jira/browse/ZOOKEEPER-1078 I'd love it if someone were to jump in and help out with this effort. (it's on trunk so it would be for the ZK 3.5.0 release)
        Hide
        Roman Shaposhnik added a comment -

        Ok, here's a bit of investigation I've done as far as the current state of tweakability of deps is concerned:

        1. ant projects (* marks the one that need to be made tweakable):
        1.1. zookeeper – no explicit dependencies on other stack components
        1.2. hadoop – no explicit dependencies on other stack components
        1.3. flume – no explicit dependencies on other stack components
        1.4. sqoop – depends on hadoop-core, hadoop-test, hbase
        1.5. pig – depends on hadoop-core, hadoop-test, zookeeper, hbase
        1.6. hive – quite broken

        2. maven projects
        2.1. hbase – hadoop-core, zookeeper
        2.2. mahout – hadoop-core
        2.3. oozie – quite broken
        2.4. whirr – hadoop-core

        Overall, I think we're doing pretty well. I'll try to run a custom build once BIGTOP-100
        gets integrated.

        Show
        Roman Shaposhnik added a comment - Ok, here's a bit of investigation I've done as far as the current state of tweakability of deps is concerned: 1. ant projects (* marks the one that need to be made tweakable): 1.1. zookeeper – no explicit dependencies on other stack components 1.2. hadoop – no explicit dependencies on other stack components 1.3. flume – no explicit dependencies on other stack components 1.4. sqoop – depends on hadoop-core , hadoop-test , hbase 1.5. pig – depends on hadoop-core, hadoop-test, zookeeper, hbase 1.6. hive – quite broken 2. maven projects 2.1. hbase – hadoop-core, zookeeper 2.2. mahout – hadoop-core 2.3. oozie – quite broken 2.4. whirr – hadoop-core Overall, I think we're doing pretty well. I'll try to run a custom build once BIGTOP-100 gets integrated.
        Hide
        Andrew Bayer added a comment -

        Bruno - I think Roman's talking about being able to, say, do test builds where we build a pre-release version of Hadoop, and then set all the other components to depend on that pre-release version we just built.

        I think working with the component projects to get overrideable dependency versions for those dependencies within Bigtop would be part of the right solution here - we've got that with Ant/Ivy already, and the HBase approach seems portable to the other Maven projects as well. It's not the best approach on a purely Maven level - I'm not a huge fan of the indirection of using property values for one-off dependency versions - but it's functional.

        Show
        Andrew Bayer added a comment - Bruno - I think Roman's talking about being able to, say, do test builds where we build a pre-release version of Hadoop, and then set all the other components to depend on that pre-release version we just built. I think working with the component projects to get overrideable dependency versions for those dependencies within Bigtop would be part of the right solution here - we've got that with Ant/Ivy already, and the HBase approach seems portable to the other Maven projects as well. It's not the best approach on a purely Maven level - I'm not a huge fan of the indirection of using property values for one-off dependency versions - but it's functional.
        Hide
        Bruno Mahé added a comment -

        Making sure that we can make freshly built bits available to downstream parts of the build as dependencies

        I am not sure to follow. It's very likely an upstream project has already its artifacts in a maven repository. So they will already be available downstream. If a project does not publish artifacts for maven, maybe we should focus on helping projects do that

        Making sure that we can override build dependencies of the downstream parts of the build to pick up versions that we specify
        I'm about to investigate how easy/difficult it'll be to solve both of them. Feel free to chime in.

        How far can we go? A patch would be perfect so we can separate the build responsibility from the version specification.

        As for the political side, I don't disagree with you, but this is still something to clear up

        Show
        Bruno Mahé added a comment - Making sure that we can make freshly built bits available to downstream parts of the build as dependencies I am not sure to follow. It's very likely an upstream project has already its artifacts in a maven repository. So they will already be available downstream. If a project does not publish artifacts for maven, maybe we should focus on helping projects do that Making sure that we can override build dependencies of the downstream parts of the build to pick up versions that we specify I'm about to investigate how easy/difficult it'll be to solve both of them. Feel free to chime in. How far can we go? A patch would be perfect so we can separate the build responsibility from the version specification. As for the political side, I don't disagree with you, but this is still something to clear up
        Hide
        Roman Shaposhnik added a comment -

        @Cos

        I think I might not have been as clear as I needed to be in my description – what I'm talking about is having a functionality in place that would ensure that what we build against and what we run against is the same set of versioned components. So far, we're actually dependent on the versioning of dependencies supplied by every upstream component. So lets take our .22 effort as an example – we would like to bump a Hadoop version to .22 and that's easy. Yet, things like Pig and HBase will still pull the versions of hadoop-core artifacts that were recorded in their ivy/maven properties at the time of a component release (definitely NOT .22). All of this will lead to an unfortunate situation where we are going to build Pig against 0.20.2 and execute against .22.

        @Bruno

        1. On a technical side – we have to problems to solve:

        • Making sure that we can make freshly built bits available to downstream parts of the build as dependencies
        • Making sure that we can override build dependencies of the downstream parts of the build to pick up versions that we specify
          I'm about to investigate how easy/difficult it'll be to solve both of them. Feel free to chime in.

        2. On a political side – there's a legitimate problem to be solved here. The problem being – we can NOT rely on the version
        information recorded as part of the release of leaf-node components, since different customers will ALWAY have different
        needs. E.g. Pig 0.9.0 is currently released as requiring 0.20.2. Yet there's plenty of customers who need it to be part
        of the stack that has Hadoop 0.20.20X as the basis OR Hadoop 0.21 as the basis, etc.

        Show
        Roman Shaposhnik added a comment - @Cos I think I might not have been as clear as I needed to be in my description – what I'm talking about is having a functionality in place that would ensure that what we build against and what we run against is the same set of versioned components. So far, we're actually dependent on the versioning of dependencies supplied by every upstream component. So lets take our .22 effort as an example – we would like to bump a Hadoop version to .22 and that's easy. Yet, things like Pig and HBase will still pull the versions of hadoop-core artifacts that were recorded in their ivy/maven properties at the time of a component release (definitely NOT .22). All of this will lead to an unfortunate situation where we are going to build Pig against 0.20.2 and execute against .22. @Bruno 1. On a technical side – we have to problems to solve: Making sure that we can make freshly built bits available to downstream parts of the build as dependencies Making sure that we can override build dependencies of the downstream parts of the build to pick up versions that we specify I'm about to investigate how easy/difficult it'll be to solve both of them. Feel free to chime in. 2. On a political side – there's a legitimate problem to be solved here. The problem being – we can NOT rely on the version information recorded as part of the release of leaf-node components, since different customers will ALWAY have different needs. E.g. Pig 0.9.0 is currently released as requiring 0.20.2. Yet there's plenty of customers who need it to be part of the stack that has Hadoop 0.20.20X as the basis OR Hadoop 0.21 as the basis, etc.
        Hide
        Bruno Mahé added a comment -

        Andrew> What if we make sure each project does not hardcode versions but uses properties which define their versions for their dependencies (similar to what HBase does here https://github.com/apache/hbase/blob/trunk/pom.xml )? We could then override them.

        Konstantin> I believe the goal would be more about ensuring a coherent set of dependencies and ensuring that inter-dependencies between projects are correct and respected. For instance, when I build flume packages, maven/ivy would pull the same versions of zookeeper we distribute and not some other arbitrary one which may or may not be compatible. Maybe Roman could clarify a little bit more?

        We should also check with mentors and Apache people if it would be allowed. Because of what has been decided during the incubation process, we cannot patch for security or build issues. And overriding dependencies versions might make us create artifacts not based on what was voted on for each project (if we want to be strict about it, which I don't really want to).
        Furthermore, we would loose all the testing done by each of these projects. So we would have the additional burden of proving these changes wouldn't break anything.

        But this seems to be something helpful and worthy to pursue or at least to think about.

        Show
        Bruno Mahé added a comment - Andrew> What if we make sure each project does not hardcode versions but uses properties which define their versions for their dependencies (similar to what HBase does here https://github.com/apache/hbase/blob/trunk/pom.xml )? We could then override them. Konstantin> I believe the goal would be more about ensuring a coherent set of dependencies and ensuring that inter-dependencies between projects are correct and respected. For instance, when I build flume packages, maven/ivy would pull the same versions of zookeeper we distribute and not some other arbitrary one which may or may not be compatible. Maybe Roman could clarify a little bit more? We should also check with mentors and Apache people if it would be allowed. Because of what has been decided during the incubation process, we cannot patch for security or build issues. And overriding dependencies versions might make us create artifacts not based on what was voted on for each project (if we want to be strict about it, which I don't really want to). Furthermore, we would loose all the testing done by each of these projects. So we would have the additional burden of proving these changes wouldn't break anything. But this seems to be something helpful and worthy to pursue or at least to think about.
        Hide
        Konstantin Boudnik added a comment - - edited

        Such on-the fly parametrization of dependencies would be disastrous IMO for it will allow to create stacks with pretty much unknown set of dependent artifacts.

        Show
        Konstantin Boudnik added a comment - - edited Such on-the fly parametrization of dependencies would be disastrous IMO for it will allow to create stacks with pretty much unknown set of dependent artifacts.
        Hide
        Andrew Bayer added a comment -

        This is actually pretty tough to do without patching the underlying components - I think we may be able to get away with passing properties to Ant/Ivy builds to force new versions for dependencies without actually editing any files, but off the top of my head, I'm not sure there's a way to do that with Maven.

        Show
        Andrew Bayer added a comment - This is actually pretty tough to do without patching the underlying components - I think we may be able to get away with passing properties to Ant/Ivy builds to force new versions for dependencies without actually editing any files, but off the top of my head, I'm not sure there's a way to do that with Maven.

          People

          • Assignee:
            Unassigned
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development