Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.4, 1.5.0
    • Fix Version/s: 1.4.5, 1.5.1
    • Component/s: None
    • Labels:
      None

      Description

      now that Hadoop 2 has a GA release, update the Hadoop 2.0 profile to target it.

        Activity

        Sean Busbey created issue -
        Sean Busbey made changes -
        Field Original Value New Value
        Assignee Sean Busbey [ busbey ]
        Hide
        Sean Busbey added a comment -

        Patch that alters hadoop 2 profile to set versions according to what hadoop-client for 2.2.0 pulls in.

        Patch as is applies to ACCUMULO-1790 branch, after ACCUMULO-1792. functional tests pass.

        Show
        Sean Busbey added a comment - Patch that alters hadoop 2 profile to set versions according to what hadoop-client for 2.2.0 pulls in. Patch as is applies to ACCUMULO-1790 branch, after ACCUMULO-1792 . functional tests pass.
        Sean Busbey made changes -
        Attachment ACCUMULO-1793.1.patch.txt [ 12614214 ]
        Sean Busbey made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        ASF subversion and git services added a comment -

        Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.4.5-SNAPSHOT from Eric Newton
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ]

        ACCUMULO-1793 make the default hadoop-2.2.0

        Show
        ASF subversion and git services added a comment - Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.4.5-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ] ACCUMULO-1793 make the default hadoop-2.2.0
        Hide
        ASF subversion and git services added a comment -

        Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.5.1-SNAPSHOT from Eric Newton
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ]

        ACCUMULO-1793 make the default hadoop-2.2.0

        Show
        ASF subversion and git services added a comment - Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.5.1-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ] ACCUMULO-1793 make the default hadoop-2.2.0
        Hide
        ASF subversion and git services added a comment -

        Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ]

        ACCUMULO-1793 make the default hadoop-2.2.0

        Show
        ASF subversion and git services added a comment - Commit 394b2b29b5b04fcc8c7d94aa5023ad376fe61114 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=394b2b2 ] ACCUMULO-1793 make the default hadoop-2.2.0
        Eric Newton made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Josh Elser added a comment - - edited

        I don't think the merge to 1.5.1-SNAPSHOT worked as intended. I'm still seeing 2.0.4-alpha as the dependency in the top-level pom.xml

        Actually, I'm really confused now. The hadoop-1.0 profile in 1.4.5-SNAPSHOT now activates 0.20.203.0? Also, it looks like we also need to update the default hadoop dependency from 1.0.4 to 1.2.1?

        It was my understanding that 1.4 was going to keep the default 0.20.* hadoop dependency, and then it would make sense to have profiles that will automatically compile against 1.2.1 and 2.2.0. Did I miss discussion on this?

        Show
        Josh Elser added a comment - - edited I don't think the merge to 1.5.1-SNAPSHOT worked as intended. I'm still seeing 2.0.4-alpha as the dependency in the top-level pom.xml Actually, I'm really confused now. The hadoop-1.0 profile in 1.4.5-SNAPSHOT now activates 0.20.203.0? Also, it looks like we also need to update the default hadoop dependency from 1.0.4 to 1.2.1? It was my understanding that 1.4 was going to keep the default 0.20.* hadoop dependency, and then it would make sense to have profiles that will automatically compile against 1.2.1 and 2.2.0. Did I miss discussion on this?
        Josh Elser made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Eric Newton added a comment - - edited

        The hadoop-1.0 profile is activated if you don't specify anything. So, the "default" version is effectively ignored. But I will make it consistent. If you want to compile against 1.2.1 you can just specify the version on the command line:

        $ mvn package -Dhadoop.version=1.2.1
        
        Show
        Eric Newton added a comment - - edited The hadoop-1.0 profile is activated if you don't specify anything. So, the "default" version is effectively ignored. But I will make it consistent. If you want to compile against 1.2.1 you can just specify the version on the command line: $ mvn package -Dhadoop.version=1.2.1
        Hide
        Josh Elser added a comment -

        Right, but the hadoop-1.0 profile sets hadoop.version=0.20.203.0. Is 0.20.x binary compatible with hadoop-1? It just seems really weird to me that hadoop-1.0 uses a 0.20.

        And yes, I think 1.4 should also depend against 1.2.1 when using a hadoop-1 profile to maintain consistency across our other active branches. I had assumed neither hadoop-1.0 nor hadoop-2.0 profiles would be active by default. I can help here if you want/like.

        Show
        Josh Elser added a comment - Right, but the hadoop-1.0 profile sets hadoop.version=0.20.203.0. Is 0.20.x binary compatible with hadoop-1? It just seems really weird to me that hadoop-1.0 uses a 0.20. And yes, I think 1.4 should also depend against 1.2.1 when using a hadoop-1 profile to maintain consistency across our other active branches. I had assumed neither hadoop-1.0 nor hadoop-2.0 profiles would be active by default. I can help here if you want/like.
        Hide
        Eric Newton added a comment -

        In my testing, accumulo compiled against 0.20.203.0 is binary compatible with hadoop 1.0, 1.2, 2.2.

        Show
        Eric Newton added a comment - In my testing, accumulo compiled against 0.20.203.0 is binary compatible with hadoop 1.0, 1.2, 2.2.
        Eric Newton made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Josh Elser added a comment -

        I don't like the fact that we now have these obtuse build profiles that don't actually line up with what they do.

        Show
        Josh Elser added a comment - I don't like the fact that we now have these obtuse build profiles that don't actually line up with what they do.
        Josh Elser made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Josh Elser added a comment -

        Reworking the profiles for dependency checking. I feel like this patch I'm providing is really dumb because it introduces another profile for no sake other than the name. I don't know what else to do because we did even have any notion of these profiles previously

        Regardless, I feel like this is still completely diverging from our mantra and I don't like it. It feels glued on to the side.

        Show
        Josh Elser added a comment - Reworking the profiles for dependency checking. I feel like this patch I'm providing is really dumb because it introduces another profile for no sake other than the name. I don't know what else to do because we did even have any notion of these profiles previously Regardless, I feel like this is still completely diverging from our mantra and I don't like it. It feels glued on to the side.
        Josh Elser made changes -
        Attachment 0001-ACCUMULO-1793-Changes-that-to-make-semi-sane-build-p.patch [ 12615986 ]
        Hide
        Sean Busbey added a comment -

        Josh Elser, Hadoop 0.20 is part of the Hadoop 1.0 line. Since our examples in the README showed building against 0.23 by specifying the hadoop 2 profile and the version (and 0.23 is part of the Hadoop 2 lineage), it didn't seem odd me to have 1.4.x default the Hadoop 1 line to 0.20. But I'm pretty used to Hadoop versions being terrible at this point.

        The binaries built against 0.20.203.0 should be binary compatible with the Hadoop 1.x releases, but should not be binary compatible with the 2.x releases. Similarly, the binaries built against 2.2.0 won't be compatible with Hadoop 1.x releases (including 0.20.203.0). Unless we backport (or create) some fancy shim work under ACCUMULO-1796.

        Which part feels glued on? Having a third profile? Since 0.20 is a Hadoop 1 version, I'm not sure that we need to. If it makes the insanity of Hadoop versions easier to follow, I don't think it hurts. We could just change the name of the Hadoop 1 profile to Hadoop-0.20 and not have 1.4 speak to 1.x.

        As an aside, I think the patch would be better suited to ACCUMULO-1795 since that's the ticket that deals with building against 0.20.203 and non-Hadoop 2 compat.

        Show
        Sean Busbey added a comment - Josh Elser , Hadoop 0.20 is part of the Hadoop 1.0 line. Since our examples in the README showed building against 0.23 by specifying the hadoop 2 profile and the version (and 0.23 is part of the Hadoop 2 lineage), it didn't seem odd me to have 1.4.x default the Hadoop 1 line to 0.20. But I'm pretty used to Hadoop versions being terrible at this point. The binaries built against 0.20.203.0 should be binary compatible with the Hadoop 1.x releases, but should not be binary compatible with the 2.x releases. Similarly, the binaries built against 2.2.0 won't be compatible with Hadoop 1.x releases (including 0.20.203.0). Unless we backport (or create) some fancy shim work under ACCUMULO-1796 . Which part feels glued on? Having a third profile? Since 0.20 is a Hadoop 1 version, I'm not sure that we need to. If it makes the insanity of Hadoop versions easier to follow, I don't think it hurts. We could just change the name of the Hadoop 1 profile to Hadoop-0.20 and not have 1.4 speak to 1.x. As an aside, I think the patch would be better suited to ACCUMULO-1795 since that's the ticket that deals with building against 0.20.203 and non-Hadoop 2 compat.
        Hide
        Sean Busbey added a comment -

        Ah, I do see an incorrect merge related to this ticket now. 1.5.1-SNAPSHOT is the only branch that doesn't show the Hadoop 2 profile having 2.2.0 as the build version (or the corresponding slf4j version).

        Show
        Sean Busbey added a comment - Ah, I do see an incorrect merge related to this ticket now. 1.5.1-SNAPSHOT is the only branch that doesn't show the Hadoop 2 profile having 2.2.0 as the build version (or the corresponding slf4j version).
        Hide
        Eric Newton added a comment -

        Sean Busbey do you know that a 1.4 compiled against hadoop 1.0 is binary incompatible with hadoop 2.0?

        I've been running it that way all day.

        Show
        Eric Newton added a comment - Sean Busbey do you know that a 1.4 compiled against hadoop 1.0 is binary incompatible with hadoop 2.0? I've been running it that way all day.
        Hide
        Sean Busbey added a comment -

        I thought I had run into the interface/class change problem. I'll try to recreate later to be sure.

        Show
        Sean Busbey added a comment - I thought I had run into the interface/class change problem. I'll try to recreate later to be sure.
        Hide
        Josh Elser added a comment -

        Hadoop 0.20 is part of the Hadoop 1.0 line

        Blarg. Ok, this is just me being stupid. Blah blah, old hadoop versioning, blah blah.

        Which part feels glued on?

        So, when we released 1.5, we started this practice of having the hadoop-1 and hadoop-2 profile which compiled against "good" versions of Hadoop for each line (read as: most recent "stable-ish"). Presently in 1.5 and beyond, this gives us hadoop-1->1.2.1 and hadoop-2->2.2.0.

        With the introduction of these profiles back in the 1.4 series, which previously only knew about 0.20.203.0, it's the same but different. Suddenly, if someone knows what we do in almost two major releases (1.5 and 1.6), and perform the same action (-Dhadoop.profile=1.0), they aren't going to compile against 1.2.1. It just doesn't sit right to me for 1) the 5th point release to suddenly have different build action than the previous 4, and 2) equivalently named build actions that do different things.

        That being said, I'm not even sure I like the patch I attached here to clarify things (pretty much why I didn't just commit it). I'm not sure what would be best. Like you said, with hadoop-0.20 being a part of "hadoop 1", but the "hadoop 1" profile using a version of hadoop in the 1.x series in 1.5, we're going to stomp on some approach we previously took. I haven't come up with anything that seems more straightforward yet.

        Show
        Josh Elser added a comment - Hadoop 0.20 is part of the Hadoop 1.0 line Blarg. Ok, this is just me being stupid. Blah blah, old hadoop versioning, blah blah. Which part feels glued on? So, when we released 1.5, we started this practice of having the hadoop-1 and hadoop-2 profile which compiled against "good" versions of Hadoop for each line (read as: most recent "stable-ish"). Presently in 1.5 and beyond, this gives us hadoop-1->1.2.1 and hadoop-2->2.2.0. With the introduction of these profiles back in the 1.4 series, which previously only knew about 0.20.203.0, it's the same but different. Suddenly, if someone knows what we do in almost two major releases (1.5 and 1.6), and perform the same action (-Dhadoop.profile=1.0), they aren't going to compile against 1.2.1. It just doesn't sit right to me for 1) the 5th point release to suddenly have different build action than the previous 4, and 2) equivalently named build actions that do different things. That being said, I'm not even sure I like the patch I attached here to clarify things (pretty much why I didn't just commit it). I'm not sure what would be best. Like you said, with hadoop-0.20 being a part of "hadoop 1", but the "hadoop 1" profile using a version of hadoop in the 1.x series in 1.5, we're going to stomp on some approach we previously took. I haven't come up with anything that seems more straightforward yet.
        Hide
        Joey Echeverria added a comment -

        Honestly, I'm surprised we advanced 1.5.1-SNAPSHOT to Hadoop 1.2.1. I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with. That means that I'd fully expect that version to change between 1.4, 1.5, and 1.6 since I would think that became fixed at the time the major version was released. I would only advance the Hadoop version if there was a known issue with what we had previously declared the lowest compatable version.

        Recommendations for which version to use at a minimum should be made in documentation, not the POM file, in my opinion. The POM file should focus on declaring compatability, not necessarily recommending a best practice.

        Show
        Joey Echeverria added a comment - Honestly, I'm surprised we advanced 1.5.1-SNAPSHOT to Hadoop 1.2.1. I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with. That means that I'd fully expect that version to change between 1.4, 1.5, and 1.6 since I would think that became fixed at the time the major version was released. I would only advance the Hadoop version if there was a known issue with what we had previously declared the lowest compatable version. Recommendations for which version to use at a minimum should be made in documentation, not the POM file, in my opinion. The POM file should focus on declaring compatability, not necessarily recommending a best practice.
        Hide
        Christopher Tubbs added a comment -

        Joey Echeverria wrote:

        I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with.

        That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility), and this makes complete sense when you're dealing with an application, like ZooKeeper, that is very rigid about being long-term backwards-compatible. I'm not sure Hadoop can be considered such an application (Accumulo certainly isn't... yet).

        The best thing, in this regard, is probably to document and test against the range of supported versions, and not to change that range of supported versions on a minor/bugfix release.... regardless of what the POM says.

        Show
        Christopher Tubbs added a comment - Joey Echeverria wrote: I always viewed the default version (and profile) to represent the lowest version that we're maintaining compatability with. That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility), and this makes complete sense when you're dealing with an application, like ZooKeeper, that is very rigid about being long-term backwards-compatible. I'm not sure Hadoop can be considered such an application (Accumulo certainly isn't... yet). The best thing, in this regard, is probably to document and test against the range of supported versions, and not to change that range of supported versions on a minor/bugfix release.... regardless of what the POM says.
        Hide
        Keith Turner added a comment -

        That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility)

        It seems thats what happened w/ ACCUMULO-1900. I only discovered that issue at runtime.

        Show
        Keith Turner added a comment - That's kind of what we've been doing with ZooKeeper (we've been sticking to the latest bugfix release of the 3.3 line, rather than jump to 3.4, so we don't accidentally use a new feature that breaks compatibility) It seems thats what happened w/ ACCUMULO-1900 . I only discovered that issue at runtime.
        Hide
        Sean Busbey added a comment -

        Josh Elser, the newly attached patch should bring 1.5.1-SNAPSHOT into alignment with the other branches for the Hadoop 2 profile. Anything else outstanding to re-close?

        Christopher Tubbs, Keith Turner, Joey Echeverria can we move discussion of which Hadoop versions to build and test against to the mailing list or a new ticket?

        Show
        Sean Busbey added a comment - Josh Elser , the newly attached patch should bring 1.5.1-SNAPSHOT into alignment with the other branches for the Hadoop 2 profile. Anything else outstanding to re-close? Christopher Tubbs , Keith Turner , Joey Echeverria can we move discussion of which Hadoop versions to build and test against to the mailing list or a new ticket?
        Sean Busbey made changes -
        Sean Busbey made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Hide
        Josh Elser added a comment - - edited

        +1 to both points

        Show
        Josh Elser added a comment - - edited +1 to both points
        Hide
        ASF subversion and git services added a comment -

        Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.5.1-SNAPSHOT from Sean Busbey
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ]

        ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0.

        Signed-off-by: Josh Elser <elserj@apache.org>

        Show
        ASF subversion and git services added a comment - Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.5.1-SNAPSHOT from Sean Busbey [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ] ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0. Signed-off-by: Josh Elser <elserj@apache.org>
        Hide
        ASF subversion and git services added a comment -

        Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.6.0-SNAPSHOT from Sean Busbey
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ]

        ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0.

        Signed-off-by: Josh Elser <elserj@apache.org>

        Show
        ASF subversion and git services added a comment - Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/1.6.0-SNAPSHOT from Sean Busbey [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ] ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0. Signed-off-by: Josh Elser <elserj@apache.org>
        Hide
        ASF subversion and git services added a comment -

        Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/master from Sean Busbey
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ]

        ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0.

        Signed-off-by: Josh Elser <elserj@apache.org>

        Show
        ASF subversion and git services added a comment - Commit 1d5ed48f94fbbac3b10d18db4148b4dae79f2880 in branch refs/heads/master from Sean Busbey [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=1d5ed48 ] ACCUMULO-1793 updates Hadoop 2 profile to match version used in other dev branches. Includes updating slf4j based on what comes with Hadoop 2.2.0. Signed-off-by: Josh Elser <elserj@apache.org>
        Josh Elser made changes -
        Attachment 0001-ACCUMULO-1793-Changes-that-to-make-semi-sane-build-p.patch [ 12615986 ]
        Josh Elser made changes -
        Fix Version/s 1.4.5 [ 12324754 ]
        Fix Version/s 1.5.1 [ 12324399 ]
        Hide
        Josh Elser added a comment -

        Thanks, Sean Busbey (and Eric Newton for his initial commits)

        Show
        Josh Elser added a comment - Thanks, Sean Busbey (and Eric Newton for his initial commits)
        Josh Elser made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        30d 20h 22m 1 Sean Busbey 18/Nov/13 18:09
        Reopened Reopened Resolved Resolved
        1h 10m 1 Eric Newton 26/Nov/13 22:50
        Resolved Resolved Reopened Reopened
        5h 5m 2 Josh Elser 27/Nov/13 02:41
        Reopened Reopened Patch Available Patch Available
        8d 17h 3m 1 Sean Busbey 05/Dec/13 19:44
        Patch Available Patch Available Resolved Resolved
        8d 10h 31m 2 Josh Elser 06/Dec/13 04:01

          People

          • Assignee:
            Sean Busbey
            Reporter:
            Sean Busbey
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development