Hadoop Common
  1. Hadoop Common
  2. HADOOP-7333

Performance improvement in PureJavaCrc32

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: performance, util
    • Labels:
      None
    • Environment:

      Linux x64

    • Hadoop Flags:
      Reviewed

      Description

      I would like to propose a small patch to

      org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)

      Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
      inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.

      If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.

      I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.

      The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.

      If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:

      414 movq R9, rsp + #24 # spill
      419 movl R9 + #12 (8-bit), RDX # int ! Field PureJavaCrc32.crc
      41d xorl R10, RDX # int

      The patch results in just one final store of the fully computed value.

      1. c7333_20110526.patch
        31 kB
        Tsz Wo Nicholas Sze
      2. HADOOP-7333.patch
        1 kB
        Eric Caspole

        Issue Links

          Activity

          Eric Caspole created issue -
          Eric Caspole made changes -
          Field Original Value New Value
          Attachment HADOOP-7333.patch [ 12480549 ]
          Hide
          Todd Lipcon added a comment -

          Hey Eric. Nice work! I'm hitting the "submit patch" button so that the QA bot will run on this. I'll also try to get a chance to reproduce your performance test results.

          Show
          Todd Lipcon added a comment - Hey Eric. Nice work! I'm hitting the "submit patch" button so that the QA bot will run on this. I'll also try to get a chance to reproduce your performance test results.
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Assignee Eric Caspole [ ecaspole ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480549/HADOOP-7333.patch
          against trunk revision 1127811.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480549/HADOOP-7333.patch against trunk revision 1127811. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          I was able to reproduce a similar performance gain on one of our test boxes (Xeon E5540 @ 2.53ghz)

          Without patch:

          Num Bytes CRC32 PureJavaCrc32
          1 15.644 167.530
          2 28.458 167.751
          4 56.510 269.064
          8 94.573 440.632
          16 144.602 458.159
          32 197.831 557.829
          64 246.404 618.492
          128 275.926 630.191
          256 294.701 646.540
          512 306.519 660.747
          1024 308.296 666.046
          2048 315.532 675.444
          4096 314.267 666.920
          8192 314.343 669.222
          16384 314.655 674.295
          32768 314.940 665.232
          65536 314.991 670.965
          131072 314.950 678.721
          262144 318.098 665.080
          524288 317.110 661.923
          1048576 315.832 672.078
          2097152 313.902 669.421
          4194304 315.350 670.617
          8388608 316.019 673.716
          16777216 316.324 657.439

          With patch:

          Num Bytes CRC32 PureJavaCrc32
          1 15.801 150.324
          2 29.117 159.762
          4 60.321 261.519
          8 100.779 423.568
          16 153.917 458.597
          32 208.510 564.227
          64 249.382 644.973
          128 281.078 714.154
          256 298.333 737.455
          512 308.425 750.980
          1024 313.149 759.572
          2048 315.489 764.234
          4096 316.663 767.175
          8192 317.716 768.538
          16384 315.930 758.694
          32768 317.824 763.277
          65536 317.246 764.298
          131072 317.927 765.582
          262144 317.217 750.438
          524288 318.005 759.085
          1048576 318.100 763.027
          2097152 318.050 762.100
          4194304 317.742 757.370
          8388608 316.887 753.327
          16777216 315.039 750.667

          Our most common use of CRC32 in DFS is with 512 byte chunk size, where we see about a 20% speedup in this test run. For big chunks, looks like about 15% - this is closer to the MR spill checksumming case.

          So, +1 pending Hudson results.

          Show
          Todd Lipcon added a comment - I was able to reproduce a similar performance gain on one of our test boxes (Xeon E5540 @ 2.53ghz) Without patch: Num Bytes CRC32 PureJavaCrc32 1 15.644 167.530 2 28.458 167.751 4 56.510 269.064 8 94.573 440.632 16 144.602 458.159 32 197.831 557.829 64 246.404 618.492 128 275.926 630.191 256 294.701 646.540 512 306.519 660.747 1024 308.296 666.046 2048 315.532 675.444 4096 314.267 666.920 8192 314.343 669.222 16384 314.655 674.295 32768 314.940 665.232 65536 314.991 670.965 131072 314.950 678.721 262144 318.098 665.080 524288 317.110 661.923 1048576 315.832 672.078 2097152 313.902 669.421 4194304 315.350 670.617 8388608 316.019 673.716 16777216 316.324 657.439 With patch: Num Bytes CRC32 PureJavaCrc32 1 15.801 150.324 2 29.117 159.762 4 60.321 261.519 8 100.779 423.568 16 153.917 458.597 32 208.510 564.227 64 249.382 644.973 128 281.078 714.154 256 298.333 737.455 512 308.425 750.980 1024 313.149 759.572 2048 315.489 764.234 4096 316.663 767.175 8192 317.716 768.538 16384 315.930 758.694 32768 317.824 763.277 65536 317.246 764.298 131072 317.927 765.582 262144 317.217 750.438 524288 318.005 759.085 1048576 318.100 763.027 2097152 318.050 762.100 4194304 317.742 757.370 8388608 316.887 753.327 16777216 315.039 750.667 Our most common use of CRC32 in DFS is with 512 byte chunk size, where we see about a 20% speedup in this test run. For big chunks, looks like about 15% - this is closer to the MR spill checksumming case. So, +1 pending Hudson results.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Eric, good observation! I just have tried your patch and got the following results. Could you also post your results? You may simply copy and paste the text output on the JIRA.

          • With patch:
            java.version = 1.6.0_10
            java.runtime.name = Java(TM) SE Runtime Environment
            java.runtime.version = 1.6.0_10-b33
            java.vm.version = 11.0-b15
            java.vm.vendor = Sun Microsystems Inc.
            java.vm.name = Java HotSpot(TM) 64-Bit Server VM
            java.vm.specification.version = 1.0
            java.specification.version = 1.6
            os.arch = amd64
            os.name = Linux
            os.version = 2.6.9-55.ELsmp

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32
          1 7.673 105.282
          2 14.646 131.429
          4 28.523 115.807
          8 51.734 281.044
          16 87.154 296.732
          32 131.368 338.979
          64 181.210 363.733
          128 219.660 376.653
          256 247.728 383.204
          512 263.704 387.144
          1024 271.574 387.665
          2048 276.787 389.429
          4096 279.420 390.218
          8192 280.808 390.630
          16384 280.895 388.867
          32768 280.641 386.111
          65536 280.908 386.008
          131072 281.049 386.089
          262144 281.117 386.124
          524288 281.133 386.130
          1048576 281.193 386.013
          2097152 281.208 385.508
          4194304 280.439 384.409
          8388608 279.023 382.025
          16777216 278.610 381.445
          • Without patch:
            java.version = 1.6.0_10
            java.runtime.name = Java(TM) SE Runtime Environment
            java.runtime.version = 1.6.0_10-b33
            java.vm.version = 11.0-b15
            java.vm.vendor = Sun Microsystems Inc.
            java.vm.name = Java HotSpot(TM) 64-Bit Server VM
            java.vm.specification.version = 1.0
            java.specification.version = 1.6
            os.arch = amd64
            os.name = Linux
            os.version = 2.6.9-55.ELsmp

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32
          1 7.812 78.613
          2 15.149 135.446
          4 28.214 144.649
          8 50.598 292.826
          16 87.112 294.373
          32 132.665 354.003
          64 179.303 387.926
          128 218.238 408.221
          256 246.192 418.157
          512 262.853 424.399
          1024 271.822 425.695
          2048 276.041 428.904
          4096 279.137 428.804
          8192 280.552 429.013
          16384 280.489 429.608
          32768 280.647 426.743
          65536 281.974 427.366
          131072 282.104 427.439
          262144 282.079 427.408
          524288 282.252 427.355
          1048576 282.310 427.171
          2097152 282.136 426.867
          4194304 280.107 425.458
          8388608 280.020 422.325
          16777216 279.599 421.762
          Show
          Tsz Wo Nicholas Sze added a comment - Eric, good observation! I just have tried your patch and got the following results. Could you also post your results? You may simply copy and paste the text output on the JIRA. With patch: java.version = 1.6.0_10 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_10-b33 java.vm.version = 11.0-b15 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.9-55.ELsmp Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 1 7.673 105.282 2 14.646 131.429 4 28.523 115.807 8 51.734 281.044 16 87.154 296.732 32 131.368 338.979 64 181.210 363.733 128 219.660 376.653 256 247.728 383.204 512 263.704 387.144 1024 271.574 387.665 2048 276.787 389.429 4096 279.420 390.218 8192 280.808 390.630 16384 280.895 388.867 32768 280.641 386.111 65536 280.908 386.008 131072 281.049 386.089 262144 281.117 386.124 524288 281.133 386.130 1048576 281.193 386.013 2097152 281.208 385.508 4194304 280.439 384.409 8388608 279.023 382.025 16777216 278.610 381.445 Without patch: java.version = 1.6.0_10 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_10-b33 java.vm.version = 11.0-b15 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.9-55.ELsmp Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 1 7.812 78.613 2 15.149 135.446 4 28.214 144.649 8 50.598 292.826 16 87.112 294.373 32 132.665 354.003 64 179.303 387.926 128 218.238 408.221 256 246.192 418.157 512 262.853 424.399 1024 271.822 425.695 2048 276.041 428.904 4096 279.137 428.804 8192 280.552 429.013 16384 280.489 429.608 32768 280.647 426.743 65536 281.974 427.366 131072 282.104 427.439 262144 282.079 427.408 524288 282.252 427.355 1048576 282.310 427.171 2097152 282.136 426.867 4194304 280.107 425.458 8388608 280.020 422.325 16777216 279.599 421.762
          Hide
          Todd Lipcon added a comment -

          Looks like Nicholas and I were doing the same thing For the record, here's my JVM info:
          java.version = 1.6.0_20
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_20-b02
          java.vm.version = 16.3-b01
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = amd64
          os.name = Linux
          os.version = 2.6.18-194.11.1.el5

          Nicholas - you good for us to commit this?

          Show
          Todd Lipcon added a comment - Looks like Nicholas and I were doing the same thing For the record, here's my JVM info: java.version = 1.6.0_20 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_20-b02 java.vm.version = 16.3-b01 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.18-194.11.1.el5 Nicholas - you good for us to commit this?
          Hide
          Eric Caspole added a comment -

          Here is an example of what I got:

          baseline:

          java.version = 1.6.0_25
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_25-b06
          java.vm.version = 20.0-b11
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = amd64
          os.name = Linux
          os.version = 2.6.35-22-generic

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32
          1 11.312 83.741
          2 21.210 160.665
          4 42.377 195.647
          8 75.407 264.901
          16 124.451 291.424
          32 183.565 376.901
          64 245.988 424.204
          128 281.473 436.240
          256 317.394 458.765
          512 335.792 479.419
          1024 347.709 473.562
          2048 351.318 476.762
          4096 351.745 485.425
          8192 357.670 479.614
          16384 355.637 481.792
          32768 357.449 483.500
          65536 358.486 472.664
          131072 355.985 483.490
          262144 361.117 483.599
          524288 357.867 468.159
          1048576 355.371 476.476
          2097152 355.238 476.458
          4194304 354.549 471.960
          8388608 349.739 465.464
          16777216 347.075 461.695

          with patch:

          java.version = 1.6.0_25
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_25-b06
          java.vm.version = 20.0-b11
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = amd64
          os.name = Linux
          os.version = 2.6.35-22-generic

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32
          1 11.388 70.238
          2 21.377 140.269
          4 42.950 195.840
          8 76.818 316.527
          16 126.218 336.181
          32 187.139 407.078
          64 246.038 450.022
          128 283.940 463.666
          256 315.997 484.709
          512 337.407 492.866
          1024 346.651 497.357
          2048 349.376 507.337
          4096 356.193 497.322
          8192 356.303 506.458
          16384 353.121 503.858
          32768 351.595 503.809
          65536 358.412 500.009
          131072 356.675 503.672
          262144 356.723 501.896
          524288 357.432 497.297
          1048576 349.544 500.216
          2097152 350.197 500.098
          4194304 350.040 497.357
          8388608 348.890 477.051
          16777216 344.792 484.409
          Show
          Eric Caspole added a comment - Here is an example of what I got: baseline: java.version = 1.6.0_25 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_25-b06 java.vm.version = 20.0-b11 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.35-22-generic Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 1 11.312 83.741 2 21.210 160.665 4 42.377 195.647 8 75.407 264.901 16 124.451 291.424 32 183.565 376.901 64 245.988 424.204 128 281.473 436.240 256 317.394 458.765 512 335.792 479.419 1024 347.709 473.562 2048 351.318 476.762 4096 351.745 485.425 8192 357.670 479.614 16384 355.637 481.792 32768 357.449 483.500 65536 358.486 472.664 131072 355.985 483.490 262144 361.117 483.599 524288 357.867 468.159 1048576 355.371 476.476 2097152 355.238 476.458 4194304 354.549 471.960 8388608 349.739 465.464 16777216 347.075 461.695 with patch: java.version = 1.6.0_25 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_25-b06 java.vm.version = 20.0-b11 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.35-22-generic Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 1 11.388 70.238 2 21.377 140.269 4 42.950 195.840 8 76.818 316.527 16 126.218 336.181 32 187.139 407.078 64 246.038 450.022 128 283.940 463.666 256 315.997 484.709 512 337.407 492.866 1024 346.651 497.357 2048 349.376 507.337 4096 356.193 497.322 8192 356.303 506.458 16384 353.121 503.858 32768 351.595 503.809 65536 358.412 500.009 131072 356.675 503.672 262144 356.723 501.896 524288 357.432 497.297 1048576 349.544 500.216 2097152 350.197 500.098 4194304 350.040 497.357 8388608 348.890 477.051 16777216 344.792 484.409
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Looks like Nicholas and I were doing the same thing ...

          Todd, this reminds me the fun when we were doing HADOOP-6166 and HADOOP-6148.

          Eric, thanks for joining us.

          > So, +1 pending Hudson results.

          +1 patch looks good. Let me run some more tests before committing this.

          Show
          Tsz Wo Nicholas Sze added a comment - > Looks like Nicholas and I were doing the same thing ... Todd, this reminds me the fun when we were doing HADOOP-6166 and HADOOP-6148 . Eric, thanks for joining us. > So, +1 pending Hudson results. +1 patch looks good. Let me run some more tests before committing this.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Tried 6 cases: three different machines, two JVMs (the latest 1.6.0_25-b06 and an earlier version) for each machine. The patch always improves the performance on the latest JVM (1.6.0_25-b06) but degrades the performance on some earlier JVMs.

          In the table below, CRC32 is java.util.zip.CRC32, PureJavaCrc32 is the current code in trunk and H7333 is the current code with the patch .


          1.1) Linux 2.6.9-55.ELsmp - Java 1.6.0_10

          java.version = 1.6.0_10
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_10-b33
          java.vm.version = 11.0-b15
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = amd64
          os.name = Linux
          os.version = 2.6.9-55.ELsmp

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32 H7333
          1 7.842 72.924 82.086
          2 15.270 106.023 121.040
          4 27.828 103.828 119.558
          8 52.146 263.903 253.742
          16 85.452 283.530 280.151
          32 130.481 343.833 325.426
          64 177.955 380.677 354.599
          128 217.593 404.632 372.172
          256 243.913 416.707 379.768
          512 261.980 423.663 384.451
          1024 271.463 425.220 386.562
          2048 276.924 429.696 389.164
          4096 279.536 430.665 390.104
          8192 280.603 431.043 390.557
          16384 281.350 431.056 389.888
          32768 281.492 427.595 387.046
          65536 281.838 426.990 385.032
          131072 281.953 427.164 386.727
          262144 282.085 427.328 387.082
          524288 282.137 427.503 387.093
          1048576 282.076 427.465 387.000
          2097152 282.106 427.219 386.776
          4194304 281.215 425.642 385.575
          8388608 280.048 422.767 383.008
          16777216 279.671 422.120 382.612

          1.2) Linux 2.6.9-55.ELsmp - Java 1.6.0_25-b06

          java.version = 1.6.0_25
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_25-b06
          java.vm.version = 20.0-b11
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = amd64
          os.name = Linux
          os.version = 2.6.9-55.ELsmp

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32 H7333
          1 7.408 105.707 91.593
          2 14.466 166.277 148.577
          4 27.889 195.146 202.841
          8 50.039 248.775 248.598
          16 84.972 329.594 335.569
          32 130.063 394.096 403.355
          64 178.457 436.276 448.967
          128 217.633 461.647 475.962
          256 245.799 474.164 490.574
          512 262.378 482.075 498.107
          1024 270.899 482.538 498.268
          2048 276.426 486.091 502.043
          4096 279.105 487.801 504.114
          8192 278.792 488.647 504.989
          16384 281.229 488.611 505.019
          32768 281.292 485.426 501.475
          65536 281.670 485.282 499.889
          131072 280.752 483.555 499.573
          262144 280.870 483.588 499.616
          524288 280.863 483.364 499.629
          1048576 280.913 483.524 499.278
          2097152 280.552 483.234 499.226
          4194304 280.172 481.426 497.313
          8388608 278.749 477.359 492.912
          16777216 278.458 476.276 492.101

          2.1) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_05-b13

          java.version = 1.6.0_05
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_05-b13
          java.vm.version = 10.0-b19
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = i386
          os.name = Linux
          os.version = 2.6.18-53.1.13.el5

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32 H7333
          1 9.705 111.072 111.346
          2 18.016 138.367 137.494
          4 34.638 213.342 226.139
          8 62.939 221.194 239.386
          16 107.019 289.123 307.963
          32 164.449 368.337 378.862
          64 224.498 425.263 423.380
          128 275.259 463.062 449.381
          256 309.731 485.426 465.417
          512 331.076 497.375 473.646
          1024 342.002 499.592 475.978
          2048 349.365 504.120 479.042
          4096 352.857 506.511 480.351
          8192 353.812 507.538 474.044
          16384 351.864 507.212 480.870
          32768 351.206 499.854 479.526
          65536 352.413 492.054 480.082
          131072 348.713 499.692 480.194
          262144 353.375 491.986 480.235
          524288 353.703 499.682 480.275
          1048576 353.387 499.606 480.259
          2097152 352.382 491.653 480.062
          4194304 352.810 499.002 479.758
          8388608 351.093 493.473 475.056
          16777216 350.968 492.484 473.960

          2.2) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_25-b06

          java.version = 1.6.0_25
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_25-b06
          java.vm.version = 20.0-b11
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = amd64
          os.name = Linux
          os.version = 2.6.18-53.1.13.el5

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32 H7333
          1 8.623 131.418 108.732
          2 17.125 195.779 179.235
          4 31.958 239.181 250.013
          8 56.425 296.820 306.358
          16 96.881 397.425 406.786
          32 154.939 475.686 499.760
          64 211.171 526.751 556.665
          128 267.582 559.062 591.984
          256 305.220 575.348 613.347
          512 326.476 583.960 623.138
          1024 338.115 585.560 624.430
          2048 345.509 589.541 628.801
          4096 349.223 591.533 630.996
          8192 351.142 592.521 632.064
          16384 352.057 592.771 628.888
          32768 351.449 588.814 627.688
          65536 352.759 588.454 627.405
          131072 352.914 588.611 627.559
          262144 352.735 588.665 627.592
          524288 353.095 588.660 627.592
          1048576 353.145 588.571 627.485
          2097152 353.140 588.262 627.276
          4194304 352.952 587.857 626.714
          8388608 350.475 581.068 618.994
          16777216 349.872 579.330 617.299

          3.1) Windows XP - Java 1.6.0_23-b05

          java.version = 1.6.0_23
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_23-b05
          java.vm.version = 19.0-b09
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) Client VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = x86
          os.name = Windows XP
          os.version = 5.1

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32 H7333
          1 4.276 62.669 64.448
          2 7.978 102.239 109.401
          4 16.730 132.354 147.217
          8 31.439 182.390 195.586
          16 54.836 219.618 234.732
          32 87.627 238.995 270.211
          64 129.170 253.900 304.123
          128 170.240 278.521 321.418
          256 211.477 286.423 341.665
          512 265.165 304.382 279.309
          1024 282.990 249.336 350.560
          2048 243.438 307.147 353.229
          4096 294.114 309.688 281.155
          8192 301.212 280.119 334.366
          16384 243.790 290.962 344.765
          32768 292.988 295.948 272.812
          65536 290.493 282.761 348.932
          131072 250.337 301.592 352.376
          262144 302.263 307.579 285.689
          524288 304.144 306.469 331.565
          1048576 289.816 205.993 350.036
          2097152 295.914 289.658 285.215
          4194304 270.211 300.434 295.595
          8388608 236.343 276.725 339.176
          16777216 291.456 295.043 311.783

          3.2) Windows XP - Java 1.6.0_25-b06

          java.version = 1.6.0_25
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_25-b06
          java.vm.version = 20.0-b11
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) Client VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = x86
          os.name = Windows XP
          os.version = 5.1

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32 H7333
          1 3.868 63.751 68.806
          2 7.819 98.380 115.517
          4 15.446 124.773 164.977
          8 29.635 174.449 237.747
          16 55.030 196.089 290.992
          32 94.131 244.999 325.735
          64 143.858 266.674 346.240
          128 192.723 330.168 281.406
          256 237.439 332.342 362.283
          512 216.150 333.393 364.037
          1024 273.192 337.913 282.942
          2048 293.097 336.928 368.440
          4096 300.675 260.551 368.051
          8192 304.412 338.468 368.042
          16384 232.549 338.616 367.385
          32768 305.303 334.487 286.756
          65536 306.435 331.820 356.131
          131072 274.592 292.947 363.439
          262144 304.008 336.345 285.965
          524288 306.601 328.582 367.409
          1048576 302.045 251.833 367.058
          2097152 296.679 333.299 362.986
          4194304 243.700 332.293 344.360
          8388608 302.936 332.315 303.883
          16777216 301.516 329.897 362.755
          Show
          Tsz Wo Nicholas Sze added a comment - Tried 6 cases: three different machines, two JVMs (the latest 1.6.0_25-b06 and an earlier version) for each machine. The patch always improves the performance on the latest JVM (1.6.0_25-b06) but degrades the performance on some earlier JVMs. In the table below, CRC32 is java.util.zip.CRC32 , PureJavaCrc32 is the current code in trunk and H7333 is the current code with the patch . 1.1) Linux 2.6.9-55.ELsmp - Java 1.6.0_10 java.version = 1.6.0_10 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_10-b33 java.vm.version = 11.0-b15 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.9-55.ELsmp Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 H7333 1 7.842 72.924 82.086 2 15.270 106.023 121.040 4 27.828 103.828 119.558 8 52.146 263.903 253.742 16 85.452 283.530 280.151 32 130.481 343.833 325.426 64 177.955 380.677 354.599 128 217.593 404.632 372.172 256 243.913 416.707 379.768 512 261.980 423.663 384.451 1024 271.463 425.220 386.562 2048 276.924 429.696 389.164 4096 279.536 430.665 390.104 8192 280.603 431.043 390.557 16384 281.350 431.056 389.888 32768 281.492 427.595 387.046 65536 281.838 426.990 385.032 131072 281.953 427.164 386.727 262144 282.085 427.328 387.082 524288 282.137 427.503 387.093 1048576 282.076 427.465 387.000 2097152 282.106 427.219 386.776 4194304 281.215 425.642 385.575 8388608 280.048 422.767 383.008 16777216 279.671 422.120 382.612 1.2) Linux 2.6.9-55.ELsmp - Java 1.6.0_25-b06 java.version = 1.6.0_25 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_25-b06 java.vm.version = 20.0-b11 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.9-55.ELsmp Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 H7333 1 7.408 105.707 91.593 2 14.466 166.277 148.577 4 27.889 195.146 202.841 8 50.039 248.775 248.598 16 84.972 329.594 335.569 32 130.063 394.096 403.355 64 178.457 436.276 448.967 128 217.633 461.647 475.962 256 245.799 474.164 490.574 512 262.378 482.075 498.107 1024 270.899 482.538 498.268 2048 276.426 486.091 502.043 4096 279.105 487.801 504.114 8192 278.792 488.647 504.989 16384 281.229 488.611 505.019 32768 281.292 485.426 501.475 65536 281.670 485.282 499.889 131072 280.752 483.555 499.573 262144 280.870 483.588 499.616 524288 280.863 483.364 499.629 1048576 280.913 483.524 499.278 2097152 280.552 483.234 499.226 4194304 280.172 481.426 497.313 8388608 278.749 477.359 492.912 16777216 278.458 476.276 492.101 2.1) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_05-b13 java.version = 1.6.0_05 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_05-b13 java.vm.version = 10.0-b19 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = i386 os.name = Linux os.version = 2.6.18-53.1.13.el5 Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 H7333 1 9.705 111.072 111.346 2 18.016 138.367 137.494 4 34.638 213.342 226.139 8 62.939 221.194 239.386 16 107.019 289.123 307.963 32 164.449 368.337 378.862 64 224.498 425.263 423.380 128 275.259 463.062 449.381 256 309.731 485.426 465.417 512 331.076 497.375 473.646 1024 342.002 499.592 475.978 2048 349.365 504.120 479.042 4096 352.857 506.511 480.351 8192 353.812 507.538 474.044 16384 351.864 507.212 480.870 32768 351.206 499.854 479.526 65536 352.413 492.054 480.082 131072 348.713 499.692 480.194 262144 353.375 491.986 480.235 524288 353.703 499.682 480.275 1048576 353.387 499.606 480.259 2097152 352.382 491.653 480.062 4194304 352.810 499.002 479.758 8388608 351.093 493.473 475.056 16777216 350.968 492.484 473.960 2.2) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_25-b06 java.version = 1.6.0_25 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_25-b06 java.vm.version = 20.0-b11 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 2.6.18-53.1.13.el5 Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 H7333 1 8.623 131.418 108.732 2 17.125 195.779 179.235 4 31.958 239.181 250.013 8 56.425 296.820 306.358 16 96.881 397.425 406.786 32 154.939 475.686 499.760 64 211.171 526.751 556.665 128 267.582 559.062 591.984 256 305.220 575.348 613.347 512 326.476 583.960 623.138 1024 338.115 585.560 624.430 2048 345.509 589.541 628.801 4096 349.223 591.533 630.996 8192 351.142 592.521 632.064 16384 352.057 592.771 628.888 32768 351.449 588.814 627.688 65536 352.759 588.454 627.405 131072 352.914 588.611 627.559 262144 352.735 588.665 627.592 524288 353.095 588.660 627.592 1048576 353.145 588.571 627.485 2097152 353.140 588.262 627.276 4194304 352.952 587.857 626.714 8388608 350.475 581.068 618.994 16777216 349.872 579.330 617.299 3.1) Windows XP - Java 1.6.0_23-b05 java.version = 1.6.0_23 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_23-b05 java.vm.version = 19.0-b09 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) Client VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = x86 os.name = Windows XP os.version = 5.1 Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 H7333 1 4.276 62.669 64.448 2 7.978 102.239 109.401 4 16.730 132.354 147.217 8 31.439 182.390 195.586 16 54.836 219.618 234.732 32 87.627 238.995 270.211 64 129.170 253.900 304.123 128 170.240 278.521 321.418 256 211.477 286.423 341.665 512 265.165 304.382 279.309 1024 282.990 249.336 350.560 2048 243.438 307.147 353.229 4096 294.114 309.688 281.155 8192 301.212 280.119 334.366 16384 243.790 290.962 344.765 32768 292.988 295.948 272.812 65536 290.493 282.761 348.932 131072 250.337 301.592 352.376 262144 302.263 307.579 285.689 524288 304.144 306.469 331.565 1048576 289.816 205.993 350.036 2097152 295.914 289.658 285.215 4194304 270.211 300.434 295.595 8388608 236.343 276.725 339.176 16777216 291.456 295.043 311.783 3.2) Windows XP - Java 1.6.0_25-b06 java.version = 1.6.0_25 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_25-b06 java.vm.version = 20.0-b11 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) Client VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = x86 os.name = Windows XP os.version = 5.1 Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 H7333 1 3.868 63.751 68.806 2 7.819 98.380 115.517 4 15.446 124.773 164.977 8 29.635 174.449 237.747 16 55.030 196.089 290.992 32 94.131 244.999 325.735 64 143.858 266.674 346.240 128 192.723 330.168 281.406 256 237.439 332.342 362.283 512 216.150 333.393 364.037 1024 273.192 337.913 282.942 2048 293.097 336.928 368.440 4096 300.675 260.551 368.051 8192 304.412 338.468 368.042 16384 232.549 338.616 367.385 32768 305.303 334.487 286.756 65536 306.435 331.820 356.131 131072 274.592 292.947 363.439 262144 304.008 336.345 285.965 524288 306.601 328.582 367.409 1048576 302.045 251.833 367.058 2097152 296.679 333.299 362.986 4194304 243.700 332.293 344.360 8388608 302.936 332.315 303.883 16777216 301.516 329.897 362.755
          Hide
          Tsz Wo Nicholas Sze added a comment -

          c7333_20110526.patch: the codes I used for testing. See if anyone wants to try it.

          Show
          Tsz Wo Nicholas Sze added a comment - c7333_20110526.patch: the codes I used for testing. See if anyone wants to try it.
          Tsz Wo Nicholas Sze made changes -
          Attachment c7333_20110526.patch [ 12480587 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Nicholas - you good for us to commit this?

          Let's wait til for a day. We should commit it unless something new is found.

          Show
          Tsz Wo Nicholas Sze added a comment - > Nicholas - you good for us to commit this? Let's wait til for a day. We should commit it unless something new is found.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480587/c7333_20110526.patch
          against trunk revision 1128003.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480587/c7333_20110526.patch against trunk revision 1128003. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Nice work checking this out on prior JVMs. I wonder if you use -XX:+AggressiveOpts on the earlier JVM if it would have the same performance increases?

          Show
          Todd Lipcon added a comment - Nice work checking this out on prior JVMs. I wonder if you use -XX:+AggressiveOpts on the earlier JVM if it would have the same performance increases?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... I wonder if you use -XX:+AggressiveOpts on the earlier JVM if it would have the same performance increases?

          By comparing the following with (2.1), it seems both columns have some better numbers but the patch still degrades the performance.


          2.3) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_05-b13 with -XX:+AggressiveOpts

          $ java -XX:+AggressiveOpts -cp hadoop-common-0.23.0-SNAPSHOT.jar:hadoop-common-test-0.23.0-SNAPSHOT.jar org.apache.hadoop.util.TestPureJavaCrc32\$PerformanceTest
          java.version = 1.6.0_05
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_05-b13
          java.vm.version = 10.0-b19
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.name = Java HotSpot(TM) Server VM
          java.vm.specification.version = 1.0
          java.specification.version = 1.6
          os.arch = i386
          os.name = Linux
          os.version = 2.6.18-53.1.13.el5

          Performance Table (The unit is MB/sec)

          Num Bytes CRC32 PureJavaCrc32 H7333
          1 9.699 111.729 108.432
          2 18.007 139.071 138.762
          4 35.060 217.921 223.770
          8 64.308 220.957 241.875
          16 108.495 290.035 308.596
          32 166.368 365.532 379.159
          64 226.203 424.410 423.369
          128 275.945 463.763 449.295
          256 311.077 485.665 465.597
          512 331.712 498.074 473.644
          1024 340.871 504.308 475.610
          2048 346.745 507.910 478.512
          4096 351.109 508.981 480.016
          8192 352.892 509.891 480.836
          16384 350.394 508.394 480.766
          32768 350.816 500.573 479.659
          65536 348.351 500.105 473.086
          131072 352.889 499.988 480.385
          262144 352.796 499.779 480.421
          524288 353.542 499.895 480.432
          1048576 353.364 499.754 480.367
          2097152 352.447 499.452 480.190
          4194304 352.423 499.187 479.956
          8388608 351.037 493.194 475.243
          16777216 350.760 492.260 474.271
          Show
          Tsz Wo Nicholas Sze added a comment - > ... I wonder if you use -XX:+AggressiveOpts on the earlier JVM if it would have the same performance increases? By comparing the following with (2.1), it seems both columns have some better numbers but the patch still degrades the performance. 2.3) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_05-b13 with -XX:+AggressiveOpts $ java -XX:+AggressiveOpts -cp hadoop-common-0.23.0-SNAPSHOT.jar:hadoop-common-test-0.23.0-SNAPSHOT.jar org.apache.hadoop.util.TestPureJavaCrc32\$PerformanceTest java.version = 1.6.0_05 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_05-b13 java.vm.version = 10.0-b19 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = i386 os.name = Linux os.version = 2.6.18-53.1.13.el5 Performance Table (The unit is MB/sec) Num Bytes CRC32 PureJavaCrc32 H7333 1 9.699 111.729 108.432 2 18.007 139.071 138.762 4 35.060 217.921 223.770 8 64.308 220.957 241.875 16 108.495 290.035 308.596 32 166.368 365.532 379.159 64 226.203 424.410 423.369 128 275.945 463.763 449.295 256 311.077 485.665 465.597 512 331.712 498.074 473.644 1024 340.871 504.308 475.610 2048 346.745 507.910 478.512 4096 351.109 508.981 480.016 8192 352.892 509.891 480.836 16384 350.394 508.394 480.766 32768 350.816 500.573 479.659 65536 348.351 500.105 473.086 131072 352.889 499.988 480.385 262144 352.796 499.779 480.421 524288 353.542 499.895 480.432 1048576 353.364 499.754 480.367 2097152 352.447 499.452 480.190 4194304 352.423 499.187 479.956 8388608 351.037 493.194 475.243 16777216 350.760 492.260 474.271
          Hide
          Todd Lipcon added a comment -

          So, I guess the question is: are we OK with trading off a 5-10% performance hit on older JVMs for a 10-20% performance gain on newer ones?

          I think yes.

          Show
          Todd Lipcon added a comment - So, I guess the question is: are we OK with trading off a 5-10% performance hit on older JVMs for a 10-20% performance gain on newer ones? I think yes.
          Hide
          Eric Caspole added a comment -

          I only tried the patch on 6u23 and 6u25. There have been many hotspot compiler optimizations introduced especially since 6u21 and my general feeling is that people who are still running 6u10 etc are more worried about stability in their exact environment vs performance.

          Show
          Eric Caspole added a comment - I only tried the patch on 6u23 and 6u25. There have been many hotspot compiler optimizations introduced especially since 6u21 and my general feeling is that people who are still running 6u10 etc are more worried about stability in their exact environment vs performance.
          Hide
          Scott Carey added a comment -

          In particular, hotspot had some changes in how it analyzes loops (loop predicates) and how it looks at register allocation as a result. I'm guessing that those changes are related.

          It is now as fast as the fastest C implementations on the latest JVM.

          I am a little disappointed that hotspot couldn't do this itself, but I suppose it makes sense to push changes to a member variable's memory address often because it can be read simultaneously.

          Show
          Scott Carey added a comment - In particular, hotspot had some changes in how it analyzes loops (loop predicates) and how it looks at register allocation as a result. I'm guessing that those changes are related. It is now as fast as the fastest C implementations on the latest JVM. I am a little disappointed that hotspot couldn't do this itself, but I suppose it makes sense to push changes to a member variable's memory address often because it can be read simultaneously.
          Hide
          Todd Lipcon added a comment -

          Sounds like we're all on board. I'll commit momentarily.

          Show
          Todd Lipcon added a comment - Sounds like we're all on board. I'll commit momentarily.
          Hide
          Todd Lipcon added a comment -

          Committed to trunk. Thanks, Eric!

          Show
          Todd Lipcon added a comment - Committed to trunk. Thanks, Eric!
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315569 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #626 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/626/)
          HADOOP-7333. Performance improvement in PureJavaCrc32. Contributed by Eric Caspole.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128425
          Files :

          • /hadoop/common/trunk/CHANGES.txt
          • /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #626 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/626/ ) HADOOP-7333 . Performance improvement in PureJavaCrc32. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128425 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #702 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/702/)
          HADOOP-7333. Performance improvement in PureJavaCrc32. Contributed by Eric Caspole.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128425
          Files :

          • /hadoop/common/trunk/CHANGES.txt
          • /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #702 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/702/ ) HADOOP-7333 . Performance improvement in PureJavaCrc32. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128425 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Suresh Srinivas made changes -
          Component/s performance [ 12316502 ]
          Suresh Srinivas made changes -
          Link This issue is related to HADOOP-6148 [ HADOOP-6148 ]
          Brandon Li made changes -
          Link This issue is related to HADOOP-8617 [ HADOOP-8617 ]

            People

            • Assignee:
              Eric Caspole
              Reporter:
              Eric Caspole
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development