Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0, 1.1.0
    • Component/s: performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-6148 implemented a Pure Java implementation of CRC32 which performs better than the built-in one. This issue is to make use of it in the mapred package

      1. mapreduce-782.txt
        2 kB
        Todd Lipcon
      2. MAPREDUCE-782.branch-1.patch
        1 kB
        Brandon Li

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          4m 1 Todd Lipcon 21/Jul/09 23:06
          Patch Available Patch Available Resolved Resolved
          5d 19h 53m 1 Tsz Wo Nicholas Sze 27/Jul/09 18:59
          Resolved Resolved Closed Closed
          393d 3h 14m 1 Tom White 24/Aug/10 22:14
          Arun C Murthy made changes -
          Fix Version/s 1.1.0 [ 12317960 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Hide
          Arun C Murthy added a comment -

          I merged this to branch-1.1 too.

          Show
          Arun C Murthy added a comment - I merged this to branch-1.1 too.
          Hide
          Arun C Murthy added a comment -

          Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?

          Show
          Arun C Murthy added a comment - Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?
          Siddharth Seth made changes -
          Fix Version/s 1.2.0 [ 12321661 ]
          Hide
          Siddharth Seth added a comment -

          Committed to branch-1.

          Show
          Siddharth Seth added a comment - Committed to branch-1.
          Hide
          Siddharth Seth added a comment -

          +1 for the backport. Committing...

          Show
          Siddharth Seth added a comment - +1 for the backport. Committing...
          Brandon Li made changes -
          Link This issue is related to HADOOP-8617 [ HADOOP-8617 ]
          Brandon Li made changes -
          Attachment MAPREDUCE-782.branch-1.patch [ 12537715 ]
          Suresh Srinivas made changes -
          Component/s performance [ 12316500 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/32/)
          . Use PureJavaCrc32 in SpillRecord. Contributed by Todd Lipcon

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #32 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/32/ ) . Use PureJavaCrc32 in SpillRecord. Contributed by Todd Lipcon
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Todd!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Todd!
          Hide
          Todd Lipcon added a comment -

          Failing tests are old and unrelated

          No new tests are required for this as the implementation was thoroughly tested by the patch in Common, and the code paths in which it is used are covered by existing tests.

          Show
          Todd Lipcon added a comment - Failing tests are old and unrelated No new tests are required for this as the implementation was thoroughly tested by the patch in Common, and the code paths in which it is used are covered by existing tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12414149/mapreduce-782.txt
          against trunk revision 797362.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414149/mapreduce-782.txt against trunk revision 797362. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/420/console This message is automatically generated.
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Todd Lipcon added a comment -

          Linking the issue that implemented PureJavaCrc32. Contains benchmark output, etc.

          Show
          Todd Lipcon added a comment - Linking the issue that implemented PureJavaCrc32. Contains benchmark output, etc.
          Todd Lipcon made changes -
          Link This issue relates to HADOOP-6148 [ HADOOP-6148 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment mapreduce-782.txt [ 12414149 ]
          Todd Lipcon created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development