Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.0.0-alpha2
-
Patch
Description
The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new test to compare Crc32C chunked verification implementations.
This test is based on Crc32PerformanceTest, what I have done in this test is:
1.CRC32C performance test.
2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it from this test.
3.User can choose either direct buffer or non-directly buffer to run this test manually.
4.Take use of verifyChunkedSumsByteArray for native to support non-directly native test.
The test result in my environment is:
[root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
java.version = 1.8.0_111
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.8.0_111-b14
java.vm.version = 25.111-b14
java.vm.vendor = Oracle Corporation
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.8
java.specification.version = 1.8
os.arch = amd64
os.name = Linux
os.version = 2.6.33.20
Data Length = 64 MB
Trials = 5
Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)
| bpc | #T | PureJava | Native | % diff |
|---|---|---|---|---|
| 32 | 1 | 394.0 | 4156.2 | 954.9% |
| 32 | 2 | 400.5 | 3679.7 | 818.7% |
| 32 | 4 | 401.8 | 2657.0 | 561.3% |
| 32 | 8 | 389.1 | 1633.8 | 319.9% |
| 32 | 16 | 222.2 | 1116.3 | 402.5% |
| bpc | #T | PureJava | Native | % diff |
| 64 | 1 | 465.0 | 5931.0 | 1175.5% |
| 64 | 2 | 468.8 | 1839.2 | 292.3% |
| 64 | 4 | 460.4 | 2968.3 | 544.7% |
| 64 | 8 | 452.4 | 1925.7 | 325.6% |
| 64 | 16 | 246.9 | 1291.8 | 423.3% |
| bpc | #T | PureJava | Native | % diff |
| 128 | 1 | 522.0 | 6147.8 | 1077.6% |
| 128 | 2 | 366.0 | 4758.5 | 1200.2% |
| 128 | 4 | 307.8 | 3265.1 | 960.8% |
| 128 | 8 | 283.6 | 2092.2 | 637.6% |
| 128 | 16 | 219.9 | 1226.1 | 457.6% |
| bpc | #T | PureJava | Native | % diff |
| 256 | 1 | 550.7 | 3177.6 | 477.0% |
| 256 | 2 | 538.6 | 1933.2 | 258.9% |
| 256 | 4 | 427.2 | 3278.1 | 667.3% |
| 256 | 8 | 420.8 | 2272.3 | 440.0% |
| 256 | 16 | 294.0 | 1311.2 | 346.0% |
| bpc | #T | PureJava | Native | % diff |
| 512 | 1 | 553.4 | 3690.4 | 566.9% |
| 512 | 2 | 455.6 | 4974.1 | 991.7% |
| 512 | 4 | 494.2 | 3406.4 | 589.2% |
| 512 | 8 | 431.4 | 2257.0 | 423.2% |
| 512 | 16 | 316.3 | 1272.0 | 302.2% |
| bpc | #T | PureJava | Native | % diff |
| 1024 | 1 | 566.1 | 3520.0 | 521.8% |
| 1024 | 2 | 508.7 | 4437.4 | 772.3% |
| 1024 | 4 | 520.7 | 3422.6 | 557.4% |
| 1024 | 8 | 501.8 | 2124.7 | 323.4% |
| 1024 | 16 | 340.6 | 1305.0 | 283.2% |
| bpc | #T | PureJava | Native | % diff |
| 2048 | 1 | 535.1 | 5438.5 | 916.4% |
| 2048 | 2 | 537.3 | 4668.3 | 768.8% |
| 2048 | 4 | 529.2 | 2417.2 | 356.7% |
| 2048 | 8 | 485.1 | 2249.8 | 363.8% |
| 2048 | 16 | 334.3 | 1265.7 | 278.6% |
| bpc | #T | PureJava | Native | % diff |
| 4096 | 1 | 563.0 | 7264.0 | 1190.1% |
| 4096 | 2 | 538.8 | 5681.4 | 954.4% |
| 4096 | 4 | 528.9 | 3107.6 | 487.5% |
| 4096 | 8 | 521.8 | 2168.9 | 315.6% |
| 4096 | 16 | 336.5 | 1256.5 | 273.4% |
| bpc | #T | PureJava | Native | % diff |
| 8192 | 1 | 521.1 | 6842.4 | 1213.1% |
| 8192 | 2 | 543.0 | 5510.2 | 914.8% |
| 8192 | 4 | 526.1 | 3759.5 | 614.6% |
| 8192 | 8 | 480.1 | 2254.1 | 369.6% |
| 8192 | 16 | 336.4 | 1213.3 | 260.6% |
| bpc | #T | PureJava | Native | % diff |
| 16384 | 1 | 521.7 | 3695.7 | 608.3% |
| 16384 | 2 | 540.8 | 5610.5 | 937.4% |
| 16384 | 4 | 539.1 | 3313.0 | 514.6% |
| 16384 | 8 | 518.7 | 2198.4 | 323.8% |
| 16384 | 16 | 337.2 | 1237.8 | 267.1% |
| bpc | #T | PureJava | Native | % diff |
| 32768 | 1 | 532.1 | 3696.0 | 594.6% |
| 32768 | 2 | 541.4 | 4647.4 | 758.4% |
| 32768 | 4 | 536.6 | 2325.1 | 333.3% |
| 32768 | 8 | 518.8 | 2036.9 | 292.6% |
| 32768 | 16 | 337.8 | 1232.0 | 264.7% |
| bpc | #T | PureJava | Native | % diff |
| 65536 | 1 | 527.7 | 8240.0 | 1461.4% |
| 65536 | 2 | 561.9 | 5472.3 | 873.9% |
| 65536 | 4 | 536.5 | 3725.2 | 594.4% |
| 65536 | 8 | 482.8 | 2040.2 | 322.5% |
| 65536 | 16 | 335.9 | 1240.4 | 269.3% |
Elapsed 77.4s