Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.4, 1.5.0
    • Fix Version/s: 1.4.5, 1.5.1
    • Component/s: None
    • Labels:
      None

      Description

      now that Hadoop 2 has a GA release, update the Hadoop 2.0 profile to target it.

        Activity

        Hide
        Josh Elser added a comment -

        Thanks, Sean Busbey (and Eric Newton for his initial commits)

        Show
        Josh Elser added a comment - Thanks, Sean Busbey (and Eric Newton for his initial commits)
        Hide
        ASF subversion and git services added a comment -

        Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/master from Sean Busbey
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ]

        ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0.

        Signed-off-by: Josh Elser <elserj@apache.org>

        Show
        ASF subversion and git services added a comment - Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/master from Sean Busbey [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ] ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0. Signed-off-by: Josh Elser <elserj@apache.org>
        Hide
        ASF subversion and git services added a comment -

        Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.6.0-SNAPSHOT from Sean Busbey
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ]

        ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0.

        Signed-off-by: Josh Elser <elserj@apache.org>

        Show
        ASF subversion and git services added a comment - Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.6.0-SNAPSHOT from Sean Busbey [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ] ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0. Signed-off-by: Josh Elser <elserj@apache.org>
        Hide
        ASF subversion and git services added a comment -

        Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.5.1-SNAPSHOT from Sean Busbey
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ]

        ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0.

        Signed-off-by: Josh Elser <elserj@apache.org>

        Show
        ASF subversion and git services added a comment - Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.5.1-SNAPSHOT from Sean Busbey [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ] ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0. Signed-off-by: Josh Elser <elserj@apache.org>
        Hide
        Josh Elser added a comment - - edited

        +1 to both points

        Show
        Josh Elser added a comment - - edited +1 to both points
        Hide
        Sean Busbey added a comment -

        Josh Elser, the newly attached patch should bring 1.5.1-SNAPSHOT into alignment with the other branches for the Hadoop 2 profile. Anything else outstanding to re-close?

        Christopher Tubbs, Keith Turner, Joey Echeverria can we move discussion of which Hadoop versions to build and test against to the mailing list or a new ticket?

        Show
        Sean Busbey added a comment - Josh Elser , the newly attached patch should bring 1.5.1-SNAPSHOT into alignment with the other branches for the Hadoop 2 profile. Anything else outstanding to re-close? Christopher Tubbs , Keith Turner , Joey Echeverria can we move discussion of which Hadoop versions to build and test against to the mailing list or a new ticket?
        Hide
        Keith Turner added a comment -

        That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility)

        It seems thats what happened w/ ACCUMULO-1900. I only discovered that issue at runtime.

        Show
        Keith Turner added a comment - That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility) It seems thats what happened w/ ACCUMULO-1900 . I only discovered that issue at runtime.
        Hide
        Christopher Tubbs added a comment -

        Joey Echeverria wrote:

        I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with.

        That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility), and this makes complete sense when you're dealing with an application, like ZooKeeper, that is very rigid about being long-term backwards-compatible. I'm not sure Hadoop can be considered such an application (Accumulo certainly isn't... yet).

        The best thing, in this regard, is probably to document and test against the range of supported versions, and not to change that range of supported versions on a minor/bugfix release.... regardless of what the POM says.

        Show
        Christopher Tubbs added a comment - Joey Echeverria wrote: I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with. That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility), and this makes complete sense when you're dealing with an application, like ZooKeeper, that is very rigid about being long-term backwards-compatible. I'm not sure Hadoop can be considered such an application (Accumulo certainly isn't... yet). The best thing, in this regard, is probably to document and test against the range of supported versions, and not to change that range of supported versions on a minor/bugfix release.... regardless of what the POM says.
        Hide
        Joey Echeverria added a comment -

        Honestly, I'm surprised we advanced 1.5.1-SNAPSHOT to Hadoop 1.2.1. I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with. That means that I'd fully expect that version to change between 1.4, 1.5, and 1.6 since I would think that became fixed at the time the major version was released. I would only advance the Hadoop version if there was a known issue with what we had previously declared the lowest compatable version.

        Recommendations for which version to use at a minimum should be made in documentation, not the POM file, in my opinion. The POM file should focus on declaring compatability, not necessarily recommending a best practice.

        Show
        Joey Echeverria added a comment - Honestly, I'm surprised we advanced 1.5.1-SNAPSHOT to Hadoop 1.2.1. I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with. That means that I'd fully expect that version to change between 1.4, 1.5, and 1.6 since I would think that became fixed at the time the major version was released. I would only advance the Hadoop version if there was a known issue with what we had previously declared the lowest compatable version. Recommendations for which version to use at a minimum should be made in documentation, not the POM file, in my opinion. The POM file should focus on declaring compatability, not necessarily recommending a best practice.
        Hide
        Josh Elser added a comment -

        Hadoop 0.20 is part of the Hadoop 1.0 line

        Blarg. Ok, this is just me being stupid. Blah blah, old hadoop versioning, blah blah.

        Which part feels glued on?

        So, when we released 1.5, we started this practice of having the hadoop-1 and hadoop-2 profile which compiled against "good" versions of Hadoop for each line (read as: most recent "stable-ish"). Presently in 1.5 and beyond, this gives us hadoop-1->1.2.1 and hadoop-2->2.2.0.

        With the introduction of these profiles back in the 1.4 series, which previously only knew about 0.20.203.0, it's the same but different. Suddenly, if someone knows what we do in almost two major releases (1.5 and 1.6), and perform the same action (-Dhadoop.profile=1.0), they aren't going to compile against 1.2.1. It just doesn't sit right to me for 1) the 5th point release to suddenly have different build action than the previous 4, and 2) equivalently named build actions that do different things.

        That being said, I'm not even sure I like the patch I attached here to clarify things (pretty much why I didn't just commit it). I'm not sure what would be best. Like you said, with hadoop-0.20 being a part of "hadoop 1", but the "hadoop 1" profile using a version of hadoop in the 1.x series in 1.5, we're going to stomp on some approach we previously took. I haven't come up with anything that seems more straightforward yet.

        Show
        Josh Elser added a comment - Hadoop 0.20 is part of the Hadoop 1.0 line Blarg. Ok, this is just me being stupid. Blah blah, old hadoop versioning, blah blah. Which part feels glued on? So, when we released 1.5, we started this practice of having the hadoop-1 and hadoop-2 profile which compiled against "good" versions of Hadoop for each line (read as: most recent "stable-ish"). Presently in 1.5 and beyond, this gives us hadoop-1->1.2.1 and hadoop-2->2.2.0. With the introduction of these profiles back in the 1.4 series, which previously only knew about 0.20.203.0, it's the same but different. Suddenly, if someone knows what we do in almost two major releases (1.5 and 1.6), and perform the same action (-Dhadoop.profile=1.0), they aren't going to compile against 1.2.1. It just doesn't sit right to me for 1) the 5th point release to suddenly have different build action than the previous 4, and 2) equivalently named build actions that do different things. That being said, I'm not even sure I like the patch I attached here to clarify things (pretty much why I didn't just commit it). I'm not sure what would be best. Like you said, with hadoop-0.20 being a part of "hadoop 1", but the "hadoop 1" profile using a version of hadoop in the 1.x series in 1.5, we're going to stomp on some approach we previously took. I haven't come up with anything that seems more straightforward yet.
        Hide
        Sean Busbey added a comment -

        I thought I had run into the interface/class change problem. I'll try to recreate later to be sure.

        Show
        Sean Busbey added a comment - I thought I had run into the interface/class change problem. I'll try to recreate later to be sure.
        Hide
        Eric Newton added a comment -

        Sean Busbey do you know that a 1.4 compiled against hadoop 1.0 is binary incompatible with hadoop 2.0?

        I've been running it that way all day.

        Show
        Eric Newton added a comment - Sean Busbey do you know that a 1.4 compiled against hadoop 1.0 is binary incompatible with hadoop 2.0? I've been running it that way all day.
        Hide
        Sean Busbey added a comment -

        Ah, I do see an incorrect merge related to this ticket now. 1.5.1-SNAPSHOT is the only branch that doesn't show the Hadoop 2 profile having 2.2.0 as the build version (or the corresponding slf4j version).

        Show
        Sean Busbey added a comment - Ah, I do see an incorrect merge related to this ticket now. 1.5.1-SNAPSHOT is the only branch that doesn't show the Hadoop 2 profile having 2.2.0 as the build version (or the corresponding slf4j version).
        Hide
        Sean Busbey added a comment -

        Josh Elser, Hadoop 0.20 is part of the Hadoop 1.0 line. Since our examples in the README showed building against 0.23 by specifying the hadoop 2 profile and the version (and 0.23 is part of the Hadoop 2 lineage), it didn't seem odd me to have 1.4.x default the Hadoop 1 line to 0.20. But I'm pretty used to Hadoop versions being terrible at this point.

        The binaries built against 0.20.203.0 should be binary compatible with the Hadoop 1.x releases, but should not be binary compatible with the 2.x releases. Similarly, the binaries built against 2.2.0 won't be compatible with Hadoop 1.x releases (including 0.20.203.0). Unless we backport (or create) some fancy shim work under ACCUMULO-1796.

        Which part feels glued on? Having a third profile? Since 0.20 is a Hadoop 1 version, I'm not sure that we need to. If it makes the insanity of Hadoop versions easier to follow, I don't think it hurts. We could just change the name of the Hadoop 1 profile to Hadoop-0.20 and not have 1.4 speak to 1.x.

        As an aside, I think the patch would be better suited to ACCUMULO-1795 since that's the ticket that deals with building against 0.20.203 and non-Hadoop 2 compat.

        Show
        Sean Busbey added a comment - Josh Elser , Hadoop 0.20 is part of the Hadoop 1.0 line. Since our examples in the README showed building against 0.23 by specifying the hadoop 2 profile and the version (and 0.23 is part of the Hadoop 2 lineage), it didn't seem odd me to have 1.4.x default the Hadoop 1 line to 0.20. But I'm pretty used to Hadoop versions being terrible at this point. The binaries built against 0.20.203.0 should be binary compatible with the Hadoop 1.x releases, but should not be binary compatible with the 2.x releases. Similarly, the binaries built against 2.2.0 won't be compatible with Hadoop 1.x releases (including 0.20.203.0). Unless we backport (or create) some fancy shim work under ACCUMULO-1796 . Which part feels glued on? Having a third profile? Since 0.20 is a Hadoop 1 version, I'm not sure that we need to. If it makes the insanity of Hadoop versions easier to follow, I don't think it hurts. We could just change the name of the Hadoop 1 profile to Hadoop-0.20 and not have 1.4 speak to 1.x. As an aside, I think the patch would be better suited to ACCUMULO-1795 since that's the ticket that deals with building against 0.20.203 and non-Hadoop 2 compat.
        Hide
        Josh Elser added a comment -

        Reworking the profiles for dependency checking. I feel like this patch I'm providing is really dumb because it introduces another profile for no sake other than the name. I don't know what else to do because we did even have any notion of these profiles previously

        Regardless, I feel like this is still completely diverging from our mantra and I don't like it. It feels glued on to the side.

        Show
        Josh Elser added a comment - Reworking the profiles for dependency checking. I feel like this patch I'm providing is really dumb because it introduces another profile for no sake other than the name. I don't know what else to do because we did even have any notion of these profiles previously Regardless, I feel like this is still completely diverging from our mantra and I don't like it. It feels glued on to the side.
        Hide
        Josh Elser added a comment -

        I don't like the fact that we now have these obtuse build profiles that don't actually line up with what they do.

        Show
        Josh Elser added a comment - I don't like the fact that we now have these obtuse build profiles that don't actually line up with what they do.
        Hide
        Eric Newton added a comment -

        In my testing, accumulo compiled against 0.20.203.0 is binary compatible with hadoop 1.0, 1.2, 2.2.

        Show
        Eric Newton added a comment - In my testing, accumulo compiled against 0.20.203.0 is binary compatible with hadoop 1.0, 1.2, 2.2.
        Hide
        Josh Elser added a comment -

        Right, but the hadoop-1.0 profile sets hadoop.version=0.20.203.0. Is 0.20.x binary compatible with hadoop-1? It just seems really weird to me that hadoop-1.0 uses a 0.20.

        And yes, I think 1.4 should also depend against 1.2.1 when using a hadoop-1 profile to maintain consistency across our other active branches. I had assumed neither hadoop-1.0 nor hadoop-2.0 profiles would be active by default. I can help here if you want/like.

        Show
        Josh Elser added a comment - Right, but the hadoop-1.0 profile sets hadoop.version=0.20.203.0. Is 0.20.x binary compatible with hadoop-1? It just seems really weird to me that hadoop-1.0 uses a 0.20. And yes, I think 1.4 should also depend against 1.2.1 when using a hadoop-1 profile to maintain consistency across our other active branches. I had assumed neither hadoop-1.0 nor hadoop-2.0 profiles would be active by default. I can help here if you want/like.
        Hide
        Eric Newton added a comment - - edited

        The hadoop-1.0 profile is activated if you don't specify anything. So, the "default" version is effectively ignored. But I will make it consistent. If you want to compile against 1.2.1 you can just specify the version on the command line:

        $ mvn package -Dhadoop.version=1.2.1
        
        Show
        Eric Newton added a comment - - edited The hadoop-1.0 profile is activated if you don't specify anything. So, the "default" version is effectively ignored. But I will make it consistent. If you want to compile against 1.2.1 you can just specify the version on the command line: $ mvn package -Dhadoop.version=1.2.1
        Hide
        Josh Elser added a comment - - edited

        I don't think the merge to 1.5.1-SNAPSHOT worked as intended. I'm still seeing 2.0.4-alpha as the dependency in the top-level pom.xml

        Actually, I'm really confused now. The hadoop-1.0 profile in 1.4.5-SNAPSHOT now activates 0.20.203.0? Also, it looks like we also need to update the default hadoop dependency from 1.0.4 to 1.2.1?

        It was my understanding that 1.4 was going to keep the default 0.20.* hadoop dependency, and then it would make sense to have profiles that will automatically compile against 1.2.1 and 2.2.0. Did I miss discussion on this?

        Show
        Josh Elser added a comment - - edited I don't think the merge to 1.5.1-SNAPSHOT worked as intended. I'm still seeing 2.0.4-alpha as the dependency in the top-level pom.xml Actually, I'm really confused now. The hadoop-1.0 profile in 1.4.5-SNAPSHOT now activates 0.20.203.0? Also, it looks like we also need to update the default hadoop dependency from 1.0.4 to 1.2.1? It was my understanding that 1.4 was going to keep the default 0.20.* hadoop dependency, and then it would make sense to have profiles that will automatically compile against 1.2.1 and 2.2.0. Did I miss discussion on this?
        Hide
        ASF subversion and git services added a comment -

        Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ]

        ACCUMULO-1793 make the default hadoop-2.2.0

        Show
        ASF subversion and git services added a comment - Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ] ACCUMULO-1793 make the default hadoop-2.2.0
        Hide
        ASF subversion and git services added a comment -

        Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.5.1-SNAPSHOT from Eric Newton
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ]

        ACCUMULO-1793 make the default hadoop-2.2.0

        Show
        ASF subversion and git services added a comment - Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.5.1-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ] ACCUMULO-1793 make the default hadoop-2.2.0
        Hide
        ASF subversion and git services added a comment -

        Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.4.5-SNAPSHOT from Eric Newton
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ]

        ACCUMULO-1793 make the default hadoop-2.2.0

        Show
        ASF subversion and git services added a comment - Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.4.5-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ] ACCUMULO-1793 make the default hadoop-2.2.0
        Hide
        Sean Busbey added a comment -

        Patch that alters hadoop 2 profile to set versions according to what hadoop-client for 2.2.0 pulls in.

        Patch as is applies to ACCUMULO-1790 branch, after ACCUMULO-1792. functional tests pass.

        Show
        Sean Busbey added a comment - Patch that alters hadoop 2 profile to set versions according to what hadoop-client for 2.2.0 pulls in. Patch as is applies to ACCUMULO-1790 branch, after ACCUMULO-1792 . functional tests pass.

          People

          • Assignee:
            Sean Busbey
            Reporter:
            Sean Busbey
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development