Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2141

Add an "extra data" field to Task for use by Mesos

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None

      Description

      In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.

      1. mapreduce-2141-v1.patch
        1 kB
        Matei Zaharia
      2. mapreduce-2141-v2.patch
        2 kB
        Matei Zaharia

        Activity

        Hide
        Matei Zaharia added a comment -

        Here is a patch for this issue. It gives the new field package level access using get and set methods, which should be enough for our purposes.

        Show
        Matei Zaharia added a comment - Here is a patch for this issue. It gives the new field package level access using get and set methods, which should be enough for our purposes.
        Hide
        Owen O'Malley added a comment -

        I'd suggest using either BytesWritable or Text rather than String. Other than that it looks good. Some additional JavaDoc that explained that the field was for the scheduler would be helpful too, I think.

        Show
        Owen O'Malley added a comment - I'd suggest using either BytesWritable or Text rather than String. Other than that it looks good. Some additional JavaDoc that explained that the field was for the scheduler would be helpful too, I think.
        Hide
        Arun C Murthy added a comment -

        Matei, the patch itself is small enough that it probably doesn't matter, but I'd still like to call out that this is a first for Hadoop Map-Reduce i.e. explicit support for non-Apache software such as Mesos.

        Personally I am happy to commit this, especially given it's innocuous nature - but I'd like to ensure others in the community are fine with this. Tom? Dhruba? (I see you guys are watching this.)


        Also, it might be worth-while to make this a 'context object' rather than a String.

        Show
        Arun C Murthy added a comment - Matei, the patch itself is small enough that it probably doesn't matter, but I'd still like to call out that this is a first for Hadoop Map-Reduce i.e. explicit support for non-Apache software such as Mesos. Personally I am happy to commit this, especially given it's innocuous nature - but I'd like to ensure others in the community are fine with this. Tom? Dhruba? (I see you guys are watching this.) Also, it might be worth-while to make this a 'context object' rather than a String.
        Hide
        Todd Lipcon added a comment -

        +1 for BytesWritable instead of String. Encoding is a pain in the butt.

        Show
        Todd Lipcon added a comment - +1 for BytesWritable instead of String. Encoding is a pain in the butt.
        Hide
        Matei Zaharia added a comment -

        Thanks for the comments, Owen and Arun. We actually plan to submit Mesos to the Apache incubator soon, so maybe it will become Apache software eventually! I'll get back to you with an updated patch.

        Show
        Matei Zaharia added a comment - Thanks for the comments, Owen and Arun. We actually plan to submit Mesos to the Apache incubator soon, so maybe it will become Apache software eventually! I'll get back to you with an updated patch.
        Hide
        Tom White added a comment -

        Arun, I'm happy with this (using BytesWritable) too.

        Show
        Tom White added a comment - Arun, I'm happy with this (using BytesWritable) too.
        Hide
        dhruba borthakur added a comment -

        +1 on getting this into Apache Hadoop!

        Show
        dhruba borthakur added a comment - +1 on getting this into Apache Hadoop!
        Hide
        Matei Zaharia added a comment -

        Here's an updated patch that uses BytesWritable instead.

        Show
        Matei Zaharia added a comment - Here's an updated patch that uses BytesWritable instead.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12457646/mapreduce-2141-v2.patch
        against trunk revision 1026450.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system tests framework. The patch passed system tests framework compile.

        Test results: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12457646/mapreduce-2141-v2.patch against trunk revision 1026450. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system tests framework. The patch passed system tests framework compile. Test results: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/3/console This message is automatically generated.
        Hide
        Matei Zaharia added a comment -

        The test failures seem to be unrelated to the patch. Let me know if there's anything else I need to do for it.

        Show
        Matei Zaharia added a comment - The test failures seem to be unrelated to the patch. Let me know if there's anything else I need to do for it.
        Hide
        Owen O'Malley added a comment -

        +1, looks good.

        Show
        Owen O'Malley added a comment - +1, looks good.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #536 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/536/)
        MAPREDUCE-2141. Add an "extra data" field to Task for use by Mesos.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #536 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/536/ ) MAPREDUCE-2141 . Add an "extra data" field to Task for use by Mesos.
        Hide
        Matei Zaharia added a comment -

        I've committed this. Thanks for the reviews, Owen and Arun!

        Show
        Matei Zaharia added a comment - I've committed this. Thanks for the reviews, Owen and Arun!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            Matei Zaharia
            Reporter:
            Matei Zaharia
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development