Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0, 1.1.0
    • Component/s: performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-6148 implemented a Pure Java implementation of CRC32 which performs better than the built-in one. This issue is to make use of it in the mapred package

      1. MAPREDUCE-782.branch-1.patch
        1 kB
        Brandon Li
      2. mapreduce-782.txt
        2 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Arun C Murthy added a comment -

          I merged this to branch-1.1 too.

          Show
          Arun C Murthy added a comment - I merged this to branch-1.1 too.
          Hide
          Arun C Murthy added a comment -

          Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?

          Show
          Arun C Murthy added a comment - Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?
          Hide
          Siddharth Seth added a comment -

          Committed to branch-1.

          Show
          Siddharth Seth added a comment - Committed to branch-1.
          Hide
          Siddharth Seth added a comment -

          +1 for the backport. Committing...

          Show
          Siddharth Seth added a comment - +1 for the backport. Committing...
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/32/)
          . Use PureJavaCrc32 in SpillRecord. Contributed by Todd Lipcon

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/32/ ) . Use PureJavaCrc32 in SpillRecord. Contributed by Todd Lipcon
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Todd!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Todd!
          Hide
          Todd Lipcon added a comment -

          Failing tests are old and unrelated

          No new tests are required for this as the implementation was thoroughly tested by the patch in Common, and the code paths in which it is used are covered by existing tests.

          Show
          Todd Lipcon added a comment - Failing tests are old and unrelated No new tests are required for this as the implementation was thoroughly tested by the patch in Common, and the code paths in which it is used are covered by existing tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12414149/mapreduce-782.txt
          against trunk revision 797362.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414149/mapreduce-782.txt against trunk revision 797362. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Todd Lipcon added a comment -

          Linking the issue that implemented PureJavaCrc32. Contains benchmark output, etc.

          Show
          Todd Lipcon added a comment - Linking the issue that implemented PureJavaCrc32. Contains benchmark output, etc.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development