Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4274

MapOutputBuffer should use native byte order for kvmeta

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha, 3.0.0
    • Component/s: performance, task
    • Labels:
      None
    • Target Version/s:

      Description

      I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create kvmeta as ByteBuffer.wrap(kvbuffer).asIntBuffer(). According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528357/mapreduce-4274.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2402//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2402//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528357/mapreduce-4274.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2402//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2402//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        I am not an expert on this code. I dug into it, but the code is somewhat complex, so I want to check one thing with you first before giving it a +1. kvmeta is a wrapper around kvbuffer, but it only is used to store the offsets into kvbuffer where the data is stored not the keys and values themselves. Those are written into kvbuffer bypassing kvmeta. So, even though kvbuffer is handed directly to the user supplied RawComparator code, the bytes in between the offsets given to RawComparator are the same as they were before the change. Is this analysis correct?

        Show
        Robert Joseph Evans added a comment - I am not an expert on this code. I dug into it, but the code is somewhat complex, so I want to check one thing with you first before giving it a +1. kvmeta is a wrapper around kvbuffer, but it only is used to store the offsets into kvbuffer where the data is stored not the keys and values themselves. Those are written into kvbuffer bypassing kvmeta. So, even though kvbuffer is handed directly to the user supplied RawComparator code, the bytes in between the offsets given to RawComparator are the same as they were before the change. Is this analysis correct?
        Hide
        Todd Lipcon added a comment -

        Hey Robert. Yep, that is my understanding as well. Of course I could have made a mistake But from what I know of this code (I helped review MAPREDUCE-64 a couple years back) this should be perfectly compatible.

        Show
        Todd Lipcon added a comment - Hey Robert. Yep, that is my understanding as well. Of course I could have made a mistake But from what I know of this code (I helped review MAPREDUCE-64 a couple years back) this should be perfectly compatible.
        Hide
        Robert Joseph Evans added a comment -

        Thanks, I just wanted a sanity check on that. +1. I'll merge it into trunk, and branch-2.

        Show
        Robert Joseph Evans added a comment - Thanks, I just wanted a sanity check on that. +1. I'll merge it into trunk, and branch-2.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Todd,

        I put this into trunk and branch-2.

        Show
        Robert Joseph Evans added a comment - Thanks Todd, I put this into trunk and branch-2.
        Hide
        Todd Lipcon added a comment -

        Thanks Bobby!

        Show
        Todd Lipcon added a comment - Thanks Bobby!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2277 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2277/)
        MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2277 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2277/ ) MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2350/)
        MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2350/ ) MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2296 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2296/)
        MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2296 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2296/ ) MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1053 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1053/)
        MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1053 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1053/ ) MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1087 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1087/)
        MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1087 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1087/ ) MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development