Hadoop Common
  1. Hadoop Common
  2. HADOOP-8617

backport pure Java CRC32 calculator changes to branch-1

        Details

        • Type: Bug Bug
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 1.1.0
        • Component/s: performance
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          Multiple efforts have been made gradually to improve the CRC performance in Hadoop. This JIRA is to back port these changes to branch-1, which include HADOOP-6166, HADOOP-6148, HADOOP-7333.

          The related HDFS and MAPREDUCE patches are uploaded to their original JIRAs HDFS-496 and MAPREDUCE-782.

            Issue Links

            relates to

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-6148 Implement a pure Java CRC32 calculator

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-6166 Improve PureJavaCrc32

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-7333 Performance improvement in PureJavaCrc32

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HDFS-496 Use PureJavaCrc32 in HDFS

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-782 Use PureJavaCrc32 in mapreduce spills

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Brandon Li added a comment -

              I ran teragen/terasort/teravalidate tests with the branch-1 patches from this JIRA and that of HDFS-496 and MAPREDUCE-782.

              Here is the result of test-patch with all the patches:

              BUILD SUCCESSFUL
Total time: 4 minutes 17 seconds

-1 overall.  

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 174 new Findbugs (version 2.0.0) warnings.

              The findbug warnings are not introduced by these patches.

              Show
              Brandon Li added a comment - I ran teragen/terasort/teravalidate tests with the branch-1 patches from this JIRA and that of HDFS-496 and MAPREDUCE-782 . Here is the result of test-patch with all the patches: BUILD SUCCESSFUL Total time: 4 minutes 17 seconds -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 174 new Findbugs (version 2.0.0) warnings. The findbug warnings are not introduced by these patches.
              Hide
              Permalink
              Tsz Wo Nicholas Sze added a comment -

              +1 patch looks good.

              Show
              Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
              Hide
              Permalink
              Tsz Wo Nicholas Sze added a comment -

              I have committed this. Thanks, Brandon!

              Show
              Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Brandon!
              Hide
              Permalink
              Arun C Murthy added a comment -

              I merged this to branch-1.1 too.

              Show
              Arun C Murthy added a comment - I merged this to branch-1.1 too.
              Hide
              Permalink
              Matt Foley added a comment -

              Closed upon release of Hadoop-1.1.0.

              Show
              Matt Foley added a comment - Closed upon release of Hadoop-1.1.0.

                People

                • Assignee:
                  Brandon Li
                  Reporter:
                  Brandon Li
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development