Hadoop Common
  1. Hadoop Common
  2. HADOOP-7446

Implement CRC32C native code using SSE4.2 instructions

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: native, performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Once HADOOP-7445 is implemented, we can get further performance improvements by implementing CRC32C using the hardware support available in SSE4.2. This support should be dynamically enabled based on CPU feature flags, and of course should be ifdeffed properly so that it doesn't break the build on architectures/platforms where it's not available.

      1. crc-pipeline-fix.txt
        4 kB
        Todd Lipcon
      2. hadoop-7446.txt
        7 kB
        Todd Lipcon
      3. hadoop-7446.txt
        7 kB
        Todd Lipcon
      4. hadoop-7446.txt
        8 kB
        Todd Lipcon
      5. pipelined_on_todds_patch.txt
        13 kB
        Kihwal Lee
      6. pipelined_with_todds_patch.txt
        18 kB
        Kihwal Lee
      7. pipelined-crc.patch.txt
        18 kB
        Kihwal Lee

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Patch on top of HADOOP-7444, HADOOP-7443, HADOOP-7445 in that order.

          Show
          Todd Lipcon added a comment - Patch on top of HADOOP-7444 , HADOOP-7443 , HADOOP-7445 in that order.
          Hide
          Kihwal Lee added a comment -

          It's probably better to use autoconf and automake to figure out the build target architecture.
          We could add AC_CANONICAL_HOST to the autoconf.in and have the cpu arch info passed when compiling.
          One way is to add -DCPU_ARCH=$(host_cpu) to AM_CFLAGS or AM_CPPFLAGS in Makefile.am.

          Or will it make more sense to use $OS_ARCH provided by build.xml?

          Show
          Kihwal Lee added a comment - It's probably better to use autoconf and automake to figure out the build target architecture. We could add AC_CANONICAL_HOST to the autoconf.in and have the cpu arch info passed when compiling. One way is to add -DCPU_ARCH=$(host_cpu) to AM_CFLAGS or AM_CPPFLAGS in Makefile.am. Or will it make more sense to use $OS_ARCH provided by build.xml?
          Hide
          Todd Lipcon added a comment -

          hmm, do you think the current ifdefs are insufficient/non-portable? Happy to switch to the autoconf method, but curious to know for my future reference.

          Show
          Todd Lipcon added a comment - hmm, do you think the current ifdefs are insufficient/non-portable? Happy to switch to the autoconf method, but curious to know for my future reference.
          Hide
          Kihwal Lee added a comment -

          Ignore my previous comment. As long as we say only gcc is supported, the macros you used are fine. Otherwise letting autoconf decide and drive would have been better.

          What I really wanted, but grossly missed in my comment above is letting ant determine the target platform and propagate to autoconf and make. Currently there are missing elements, but can be addressed in a separate jira. The ideal picture I am imagining is, if a build machine is properly set up, the user should be able to build and generate packages for all supported arch and platforms by simply telling one or more targets to ant.

          One last thing. Would you shed some light on this comment in the code?

          // alignment doesn't seem to help?
          
          Show
          Kihwal Lee added a comment - Ignore my previous comment. As long as we say only gcc is supported, the macros you used are fine. Otherwise letting autoconf decide and drive would have been better. What I really wanted, but grossly missed in my comment above is letting ant determine the target platform and propagate to autoconf and make. Currently there are missing elements, but can be addressed in a separate jira. The ideal picture I am imagining is, if a build machine is properly set up, the user should be able to build and generate packages for all supported arch and platforms by simply telling one or more targets to ant. One last thing. Would you shed some light on this comment in the code? // alignment doesn't seem to help?
          Hide
          Kihwal Lee added a comment -

          One last thing

          I spoke too soon. What about calling cpuid() from the library constructor and set a global var?

          Show
          Kihwal Lee added a comment - One last thing I spoke too soon. What about calling cpuid() from the library constructor and set a global var?
          Hide
          Todd Lipcon added a comment -

          Patch rebased on trunk, and taking into account Kihwal's comments above.

          I continued to use the gcc-specific macros inside an ifdef _GNUC_. If someone wants to port this optimization to work on a different compiler, that would be a good followup. I don't have anything else available to test with, here.

          Show
          Todd Lipcon added a comment - Patch rebased on trunk, and taking into account Kihwal's comments above. I continued to use the gcc-specific macros inside an ifdef _ GNUC _. If someone wants to port this optimization to work on a different compiler, that would be a good followup. I don't have anything else available to test with, here.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In the code below, rbx should be ebx.

          +  asm("pushq %%rbx\n\t" /* save %rbx */
          +      "cpuid\n\t"
          +      "movl %%ebx, %[ebx]\n\t" /* save what cpuid just put in %ebx */
          +      "popq %%rbx" : "=a"(eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" (eax_in)
          
          Show
          Tsz Wo Nicholas Sze added a comment - In the code below, rbx should be ebx. + asm( "pushq %%rbx\n\t" /* save %rbx */ + "cpuid\n\t" + "movl %%ebx, %[ebx]\n\t" /* save what cpuid just put in %ebx */ + "popq %%rbx" : "=a" (eax), [ebx] "=r" (ebx), "=c" (ecx), "=d" (edx) : "a" (eax_in)
          Hide
          Todd Lipcon added a comment -

          Why's that? rbx is the 64-bit version of ebx - we need to save off and restore the whole register, even though cpuid only writes 32 bits into it. Or am I misunderstanding this?

          Show
          Todd Lipcon added a comment - Why's that? rbx is the 64-bit version of ebx - we need to save off and restore the whole register, even though cpuid only writes 32 bits into it. Or am I misunderstanding this?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Oops, I missed that. Would rbx work on 32-bit machines?

          If yes,
          +1 patch look good. Nice work.

          Show
          Tsz Wo Nicholas Sze added a comment - Oops, I missed that. Would rbx work on 32-bit machines? If yes, +1 patch look good. Nice work.
          Hide
          Kihwal Lee added a comment -

          I don't think rbx will work on 32 bit x86 processors. It's fine for 64 bit machines running 32 bit OS/lib.
          I will post my final comment today after some testing.

          Show
          Kihwal Lee added a comment - I don't think rbx will work on 32 bit x86 processors. It's fine for 64 bit machines running 32 bit OS/lib. I will post my final comment today after some testing.
          Hide
          Todd Lipcon added a comment -

          You're right about rbx. I also found a bug where we didn't properly notate ebx as a clobbered register.

          In testing the fixes, I figure out that the current native code seems to segfault on 32-bit... I'll open a separate JIRA to fix that before we commit this one.

          Show
          Todd Lipcon added a comment - You're right about rbx. I also found a bug where we didn't properly notate ebx as a clobbered register. In testing the fixes, I figure out that the current native code seems to segfault on 32-bit... I'll open a separate JIRA to fix that before we commit this one.
          Hide
          Todd Lipcon added a comment -

          actually scratch that... I think I hadn't done a full clean build or something - once I tried to debug it, I couldn't reproduce. But if someone else can verify that "mvn -Pnative test -Dtest=TestDataChecksum" works when they set JAVA_HOME to a 32-bit JDK, that would be great. I'll upload a fixed patch momentarily

          Show
          Todd Lipcon added a comment - actually scratch that... I think I hadn't done a full clean build or something - once I tried to debug it, I couldn't reproduce. But if someone else can verify that "mvn -Pnative test -Dtest=TestDataChecksum" works when they set JAVA_HOME to a 32-bit JDK, that would be great. I'll upload a fixed patch momentarily
          Hide
          Todd Lipcon added a comment -

          New patch to fix up compilation issues on 32-bit:

          The PIC workaround is only necessary on 32-bit PIC. If you try to use it on 64-bit PIC, you end up with a subtle bug: gcc may actually allocate the "ebx" variable in the %ebx register, and then you end up getting the wrong result. So, the fix is to only do the %ebx save off to the stack in 32-bit mode. See http://www.technovelty.org/code/arch/pic-cas.html for a good explanation.

          Given this, we now only worry about %ebx and not %rbx, since on 64-bit platforms we don't have to save anything aside.

          I've tested this code via TestDataChecksum on both 32-bit and 64-bit JVMs, and also ran a short standalone test of the bulk_verify_crc32 under valgrind to check for any possible memory errors.

          Show
          Todd Lipcon added a comment - New patch to fix up compilation issues on 32-bit: The PIC workaround is only necessary on 32-bit PIC. If you try to use it on 64-bit PIC, you end up with a subtle bug: gcc may actually allocate the "ebx" variable in the %ebx register, and then you end up getting the wrong result. So, the fix is to only do the %ebx save off to the stack in 32-bit mode. See http://www.technovelty.org/code/arch/pic-cas.html for a good explanation. Given this, we now only worry about %ebx and not %rbx, since on 64-bit platforms we don't have to save anything aside. I've tested this code via TestDataChecksum on both 32-bit and 64-bit JVMs, and also ran a short standalone test of the bulk_verify_crc32 under valgrind to check for any possible memory errors.
          Hide
          Kihwal Lee added a comment -

          I wrote a simple C test program that reads a file 64K at a time and calls the bulk verification function in a loop. In zlib case, it reads 512 bytes at a time and calls the zlib API. The page cache was warmed up by running multiple times and the best numbers were picked. The following is to show roughly how they perform differently. Do not take it as an accurate analysis.

          Environment & Setup
          data file 65536000 bytes
          64 KB buffer, 512B chunk size
          GCC optimization level 2 (-O2)
          2400 MHz Intel Xeon E5530

          Results

          [Software slice-by-8]
          64 bit software 137,944,134 cycles  1140 MB/s
          32 bit software 183,254,871 cycles   858 MB/s
          
          [Hardware non-pipelined]
          64 bit hardware  53,547,822 cycles  2937 MB/s
          32 bit hardware  73,407,888 cycles  2142 MB/s
          
          [Hardware Pipelined]
          64 bit hardware  42,188,922 cycles  3728 MB/s
          32 bit hardware  49,568,661 cycles  3173 MB/s
          
          [zlib]
          32 bit          478,252,812 cycles   328 MB/s
          64 bit          490,850,076 cycles   320 MB/s
          

          I will soon post the code for the pipelined version after rebasing to your new patch. You can take a look and decide what to do.

          Show
          Kihwal Lee added a comment - I wrote a simple C test program that reads a file 64K at a time and calls the bulk verification function in a loop. In zlib case, it reads 512 bytes at a time and calls the zlib API. The page cache was warmed up by running multiple times and the best numbers were picked. The following is to show roughly how they perform differently. Do not take it as an accurate analysis. Environment & Setup data file 65536000 bytes 64 KB buffer, 512B chunk size GCC optimization level 2 (-O2) 2400 MHz Intel Xeon E5530 Results [Software slice-by-8] 64 bit software 137,944,134 cycles 1140 MB/s 32 bit software 183,254,871 cycles 858 MB/s [Hardware non-pipelined] 64 bit hardware 53,547,822 cycles 2937 MB/s 32 bit hardware 73,407,888 cycles 2142 MB/s [Hardware Pipelined] 64 bit hardware 42,188,922 cycles 3728 MB/s 32 bit hardware 49,568,661 cycles 3173 MB/s [zlib] 32 bit 478,252,812 cycles 328 MB/s 64 bit 490,850,076 cycles 320 MB/s I will soon post the code for the pipelined version after rebasing to your new patch. You can take a look and decide what to do.
          Hide
          Kihwal Lee added a comment -

          I am attaching two patches for the pipelined checksum calculation. One is the one on top of your patch and the other includes your patch.

          #define USE_PIPELINED enables it. I cross examined the crc results from slice-by-8, hardware and hardware-pipelined. I also tried 1 to 8 bytes, less than one chunk(block) to three. They all match.

          Show
          Kihwal Lee added a comment - I am attaching two patches for the pipelined checksum calculation. One is the one on top of your patch and the other includes your patch. #define USE_PIPELINED enables it. I cross examined the crc results from slice-by-8, hardware and hardware-pipelined. I also tried 1 to 8 bytes, less than one chunk(block) to three. They all match.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12490319/pipelined_with_todds_patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490319/pipelined_with_todds_patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//artifact/trunk/target/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//artifact/trunk/target/newPatchFindbugsWarningshadoop-annotations.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/36//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Hey Kihwal. I tried your pipelined CRC implementation with HBase and it's causing lots of checksum errors (no segfaults, just an incorrect mismatched checksum). I haven't had a chance to debug it yet, but given the issues I think I'll move forward committing the non-pipelined implementation and we can do the pipelined one in a follow-on JIRA?

          Show
          Todd Lipcon added a comment - Hey Kihwal. I tried your pipelined CRC implementation with HBase and it's causing lots of checksum errors (no segfaults, just an incorrect mismatched checksum). I haven't had a chance to debug it yet, but given the issues I think I'll move forward committing the non-pipelined implementation and we can do the pipelined one in a follow-on JIRA?
          Hide
          Todd Lipcon added a comment -

          Found the bug in your code – you weren't updating the data pointer when there was a remaining group of blocks past processing all the triplets. Also fixed a typo in spelling the function name. With this patch I can run the hbase test without checksum errors.

          Show
          Todd Lipcon added a comment - Found the bug in your code – you weren't updating the data pointer when there was a remaining group of blocks past processing all the triplets. Also fixed a typo in spelling the function name. With this patch I can run the hbase test without checksum errors.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12491279/crc-pipeline-fix.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/64//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491279/crc-pipeline-fix.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/64//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          It was a test and you passed. I am just kidding.
          Thanks for fixing my buggy code.

          Show
          Kihwal Lee added a comment - It was a test and you passed. I am just kidding. Thanks for fixing my buggy code.
          Hide
          Kihwal Lee added a comment -

          The new patch includes everything + a bit of clean-up. It applies to both branch-0.23 and trunk.

          Show
          Kihwal Lee added a comment - The new patch includes everything + a bit of clean-up. It applies to both branch-0.23 and trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493172/pipelined-crc.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493172/pipelined-crc.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-auth-examples.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/140//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1, will commit this patch momentarily. Thanks for the help, Kihwal.

          Show
          Todd Lipcon added a comment - +1, will commit this patch momentarily. Thanks for the help, Kihwal.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/100/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/100/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1217 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1217/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1217 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1217/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1269/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1269/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1193 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1193/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1193 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1193/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/100/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/100/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #106 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/106/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #106 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/106/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #847 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/847/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #847 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/847/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #878 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/878/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #878 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/878/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190650 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/)
          HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/ ) HADOOP-7446 . Implement CRC32C native code using SSE4.2 instructions. Contributed by Kihwal Lee and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190649 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development