Mesos
  1. Mesos
  2. MESOS-13

Port Hadoop framework to new Java API.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: java api
    • Labels:

      Description

      The supplied Hadoop patch has not been updated to work against the new Java API.

      1. hadoop-for-new-iface.patch
        105 kB
        Charles Reiss
      2. hadoop-for-new-iface.patch
        658 kB
        Charles Reiss
      3. hadoop-for-new-iface.patch
        657 kB
        Charles Reiss
      4. hadoop-mesos.patch
        79 kB
        Charles Reiss
      5. MESOS-13.patch
        6 kB
        Matei Zaharia

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2719/#review3384
        -----------------------------------------------------------

        Ship it!

        Giving this a ship-it so that (hopefully) I can close the review request since it has been committed.

        • Andy

        On 2011-11-08 23:45:31, Charles Reiss wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2719/

        -----------------------------------------------------------

        (Updated 2011-11-08 23:45:31)

        Review request for mesos.

        Summary

        -------

        How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles.

        An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch).

        This addresses bug MESOS-13.

        https://issues.apache.org/jira/browse/MESOS-13

        Diffs

        -----

        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772

        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195

        Diff: https://reviews.apache.org/r/2719/diff

        Testing

        -------

        Thanks,

        Charles

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/#review3384 ----------------------------------------------------------- Ship it! Giving this a ship-it so that (hopefully) I can close the review request since it has been committed. Andy On 2011-11-08 23:45:31, Charles Reiss wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/ ----------------------------------------------------------- (Updated 2011-11-08 23:45:31) Review request for mesos. Summary ------- How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles. An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch). This addresses bug MESOS-13 . https://issues.apache.org/jira/browse/MESOS-13 Diffs ----- frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772 frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195 Diff: https://reviews.apache.org/r/2719/diff Testing ------- Thanks, Charles
        Hide
        Matei Zaharia added a comment -

        Great, thanks.

        Show
        Matei Zaharia added a comment - Great, thanks.
        Hide
        Charles Reiss added a comment -

        Updated hadoop-mesos.patch

        Show
        Charles Reiss added a comment - Updated hadoop-mesos.patch
        Hide
        Matei Zaharia added a comment -

        I committed the patch to SVN, Charles, but then I realized that it didn't include the updated diff. Do you mind attaching a patch with both of those? I can then commit that too.

        Show
        Matei Zaharia added a comment - I committed the patch to SVN, Charles, but then I realized that it didn't include the updated diff. Do you mind attaching a patch with both of those? I can then commit that too.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2719/#review3122
        -----------------------------------------------------------

        Ship it!

        This looks great. I'm going to commit it in a sec.

        • Matei

        On 2011-11-08 23:45:31, Charles Reiss wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2719/

        -----------------------------------------------------------

        (Updated 2011-11-08 23:45:31)

        Review request for mesos.

        Summary

        -------

        How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles.

        An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch).

        This addresses bug MESOS-13.

        https://issues.apache.org/jira/browse/MESOS-13

        Diffs

        -----

        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772

        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195

        Diff: https://reviews.apache.org/r/2719/diff

        Testing

        -------

        Thanks,

        Charles

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/#review3122 ----------------------------------------------------------- Ship it! This looks great. I'm going to commit it in a sec. Matei On 2011-11-08 23:45:31, Charles Reiss wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/ ----------------------------------------------------------- (Updated 2011-11-08 23:45:31) Review request for mesos. Summary ------- How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles. An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch). This addresses bug MESOS-13 . https://issues.apache.org/jira/browse/MESOS-13 Diffs ----- frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772 frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195 Diff: https://reviews.apache.org/r/2719/diff Testing ------- Thanks, Charles
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2719/
        -----------------------------------------------------------

        (Updated 2011-11-08 23:45:31.135667)

        Review request for mesos.

        Changes
        -------

        Changed to a parallel ArrayList of lists of candidate tasks.

        Summary
        -------

        How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles.

        An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch).

        This addresses bug MESOS-13.
        https://issues.apache.org/jira/browse/MESOS-13

        Diffs (updated)


        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772
        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195

        Diff: https://reviews.apache.org/r/2719/diff

        Testing
        -------

        Thanks,

        Charles

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/ ----------------------------------------------------------- (Updated 2011-11-08 23:45:31.135667) Review request for mesos. Changes ------- Changed to a parallel ArrayList of lists of candidate tasks. Summary ------- How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles. An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch). This addresses bug MESOS-13 . https://issues.apache.org/jira/browse/MESOS-13 Diffs (updated) frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772 frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195 Diff: https://reviews.apache.org/r/2719/diff Testing ------- Thanks, Charles
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2719/#review3099
        -----------------------------------------------------------

        The patch looks good to me overall, but one issue is that you're building a HashMap with protobuf objects (OfferID) as the key in the resource offer handler. According to http://code.google.com/p/protobuf/issues/detail?id=288, hashCode() can be inconsistent for protobuf objects, and I remember running into a problem with it before. It might be better to just call toString on the IDs or even have an array of ArrayLists in the same order as the offers.

        • Matei

        On 2011-11-04 00:17:22, Charles Reiss wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2719/

        -----------------------------------------------------------

        (Updated 2011-11-04 00:17:22)

        Review request for mesos.

        Summary

        -------

        How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles.

        An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch).

        This addresses bug MESOS-13.

        https://issues.apache.org/jira/browse/MESOS-13

        Diffs

        -----

        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772

        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195

        Diff: https://reviews.apache.org/r/2719/diff

        Testing

        -------

        Thanks,

        Charles

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/#review3099 ----------------------------------------------------------- The patch looks good to me overall, but one issue is that you're building a HashMap with protobuf objects (OfferID) as the key in the resource offer handler. According to http://code.google.com/p/protobuf/issues/detail?id=288 , hashCode() can be inconsistent for protobuf objects, and I remember running into a problem with it before. It might be better to just call toString on the IDs or even have an array of ArrayLists in the same order as the offers. Matei On 2011-11-04 00:17:22, Charles Reiss wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/ ----------------------------------------------------------- (Updated 2011-11-04 00:17:22) Review request for mesos. Summary ------- How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles. An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch). This addresses bug MESOS-13 . https://issues.apache.org/jira/browse/MESOS-13 Diffs ----- frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772 frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195 Diff: https://reviews.apache.org/r/2719/diff Testing ------- Thanks, Charles
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2719/
        -----------------------------------------------------------

        Review request for mesos.

        Summary
        -------

        How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles.

        An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch).

        This addresses bug MESOS-13.
        https://issues.apache.org/jira/browse/MESOS-13

        Diffs


        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772
        frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195

        Diff: https://reviews.apache.org/r/2719/diff

        Testing
        -------

        Thanks,

        Charles

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2719/ ----------------------------------------------------------- Review request for mesos. Summary ------- How offers are presented in the Java scheduler API and how scheduler registration parameters are passed changed recently and the sample hadoop-0.20.2 framework was not updated. This patch updates the Hadoop framework so it compiles. An additional change will be needed to update the supplied .diff file (that will not be a human-readable patch). This addresses bug MESOS-13 . https://issues.apache.org/jira/browse/MESOS-13 Diffs frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/FrameworkScheduler.java f2d5772 frameworks/hadoop-0.20.2/src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java e2c1195 Diff: https://reviews.apache.org/r/2719/diff Testing ------- Thanks, Charles
        Hide
        Charles Reiss added a comment -

        This appears to be broken again after https://github.com/mesos/mesos/commit/db4b7fe632839f535aa26b963ea7c2b6d9543fe9 (which got into Apache trunk in r1195609 apparently).

        Show
        Charles Reiss added a comment - This appears to be broken again after https://github.com/mesos/mesos/commit/db4b7fe632839f535aa26b963ea7c2b6d9543fe9 (which got into Apache trunk in r1195609 apparently).
        Hide
        Matei Zaharia added a comment -

        I've committed this with some small tweaks to work with some recent API changes. Thanks Charles!

        Show
        Matei Zaharia added a comment - I've committed this with some small tweaks to work with some recent API changes. Thanks Charles!
        Hide
        Bill Zhao added a comment -

        Charles and Ben,

        I would like to get the current version of Mesos working with hadoop-0.20.2. Do you have any advice on this?

        Show
        Bill Zhao added a comment - Charles and Ben, I would like to get the current version of Mesos working with hadoop-0.20.2. Do you have any advice on this?
        Hide
        Benjamin Hindman added a comment -

        There might be a few more API changes in the version we push to Apache SVN on Saturday, I'll let you know and we can flush out the details. Again, this is awesome! Thanks!

        Show
        Benjamin Hindman added a comment - There might be a few more API changes in the version we push to Apache SVN on Saturday, I'll let you know and we can flush out the details. Again, this is awesome! Thanks!
        Hide
        Charles Reiss added a comment -

        Attached patch which doesn't also include a bunch of extraneous changes. (Oops!)

        Show
        Charles Reiss added a comment - Attached patch which doesn't also include a bunch of extraneous changes. (Oops!)
        Hide
        Charles Reiss added a comment -

        Attached slightly updated patch (rmv. FIXME comments, cleanup log messages.)

        Show
        Charles Reiss added a comment - Attached slightly updated patch (rmv. FIXME comments, cleanup log messages.)
        Hide
        Charles Reiss added a comment -

        Patch to update frameworks/hadoop-0.20.2. Could probably use better testing.

        Show
        Charles Reiss added a comment - Patch to update frameworks/hadoop-0.20.2. Could probably use better testing.

          People

          • Assignee:
            Charles Reiss
            Reporter:
            Charles Reiss
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development