Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0, 1.1.0
    • Component/s: performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-6148 implemented a Pure Java implementation of CRC32 which performs better than the built-in one. This issue is to make use of it in the mapred package

      1. MAPREDUCE-782.branch-1.patch
        1 kB
        Brandon Li
      2. mapreduce-782.txt
        2 kB
        Todd Lipcon

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment mapreduce-782.txt [ 12414149 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Link This issue relates to HADOOP-6148 [ HADOOP-6148 ]
          Hide
          Todd Lipcon added a comment -

          Linking the issue that implemented PureJavaCrc32. Contains benchmark output, etc.

          Show
          Todd Lipcon added a comment - Linking the issue that implemented PureJavaCrc32. Contains benchmark output, etc.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12414149/mapreduce-782.txt
          against trunk revision 797362.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414149/mapreduce-782.txt against trunk revision 797362. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Failing tests are old and unrelated

          No new tests are required for this as the implementation was thoroughly tested by the patch in Common, and the code paths in which it is used are covered by existing tests.

          Show
          Todd Lipcon added a comment - Failing tests are old and unrelated No new tests are required for this as the implementation was thoroughly tested by the patch in Common, and the code paths in which it is used are covered by existing tests.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Todd!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Todd!
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/32/)
          . Use PureJavaCrc32 in SpillRecord. Contributed by Todd Lipcon

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/32/ ) . Use PureJavaCrc32 in SpillRecord. Contributed by Todd Lipcon
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Suresh Srinivas made changes -
          Component/s performance [ 12316500 ]
          Brandon Li made changes -
          Attachment MAPREDUCE-782.branch-1.patch [ 12537715 ]
          Brandon Li made changes -
          Link This issue is related to HADOOP-8617 [ HADOOP-8617 ]
          Hide
          Siddharth Seth added a comment -

          +1 for the backport. Committing...

          Show
          Siddharth Seth added a comment - +1 for the backport. Committing...
          Hide
          Siddharth Seth added a comment -

          Committed to branch-1.

          Show
          Siddharth Seth added a comment - Committed to branch-1.
          Siddharth Seth made changes -
          Fix Version/s 1.2.0 [ 12321661 ]
          Hide
          Arun C Murthy added a comment -

          Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?

          Show
          Arun C Murthy added a comment - Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?
          Hide
          Arun C Murthy added a comment -

          I merged this to branch-1.1 too.

          Show
          Arun C Murthy added a comment - I merged this to branch-1.1 too.
          Arun C Murthy made changes -
          Fix Version/s 1.1.0 [ 12317960 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          4m 1 Todd Lipcon 21/Jul/09 22:06
          Patch Available Patch Available Resolved Resolved
          5d 19h 53m 1 Tsz Wo Nicholas Sze 27/Jul/09 17:59
          Resolved Resolved Closed Closed
          393d 3h 14m 1 Tom White 24/Aug/10 21:14

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development