Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14424

Add CRC32C performance test.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0-alpha2
    • 2.9.0, 3.0.0-alpha4
    • common

    Description

      The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new test to compare Crc32C chunked verification implementations.
      This test is based on Crc32PerformanceTest, what I have done in this test is:
      1.CRC32C performance test.
      2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it from this test.
      3.User can choose either direct buffer or non-directly buffer to run this test manually.
      4.Take use of verifyChunkedSumsByteArray for native to support non-directly native test.

      The test result in my environment is:
      [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
      java.version = 1.8.0_111
      java.runtime.name = Java(TM) SE Runtime Environment
      java.runtime.version = 1.8.0_111-b14
      java.vm.version = 25.111-b14
      java.vm.vendor = Oracle Corporation
      java.vm.name = Java HotSpot(TM) 64-Bit Server VM
      java.vm.specification.version = 1.8
      java.specification.version = 1.8
      os.arch = amd64
      os.name = Linux
      os.version = 2.6.33.20
      Data Length = 64 MB
      Trials = 5

      Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)

      bpc #T PureJava Native % diff
      32 1 394.0 4156.2 954.9%
      32 2 400.5 3679.7 818.7%
      32 4 401.8 2657.0 561.3%
      32 8 389.1 1633.8 319.9%
      32 16 222.2 1116.3 402.5%
      bpc #T PureJava Native % diff
      64 1 465.0 5931.0 1175.5%
      64 2 468.8 1839.2 292.3%
      64 4 460.4 2968.3 544.7%
      64 8 452.4 1925.7 325.6%
      64 16 246.9 1291.8 423.3%
      bpc #T PureJava Native % diff
      128 1 522.0 6147.8 1077.6%
      128 2 366.0 4758.5 1200.2%
      128 4 307.8 3265.1 960.8%
      128 8 283.6 2092.2 637.6%
      128 16 219.9 1226.1 457.6%
      bpc #T PureJava Native % diff
      256 1 550.7 3177.6 477.0%
      256 2 538.6 1933.2 258.9%
      256 4 427.2 3278.1 667.3%
      256 8 420.8 2272.3 440.0%
      256 16 294.0 1311.2 346.0%
      bpc #T PureJava Native % diff
      512 1 553.4 3690.4 566.9%
      512 2 455.6 4974.1 991.7%
      512 4 494.2 3406.4 589.2%
      512 8 431.4 2257.0 423.2%
      512 16 316.3 1272.0 302.2%
      bpc #T PureJava Native % diff
      1024 1 566.1 3520.0 521.8%
      1024 2 508.7 4437.4 772.3%
      1024 4 520.7 3422.6 557.4%
      1024 8 501.8 2124.7 323.4%
      1024 16 340.6 1305.0 283.2%
      bpc #T PureJava Native % diff
      2048 1 535.1 5438.5 916.4%
      2048 2 537.3 4668.3 768.8%
      2048 4 529.2 2417.2 356.7%
      2048 8 485.1 2249.8 363.8%
      2048 16 334.3 1265.7 278.6%
      bpc #T PureJava Native % diff
      4096 1 563.0 7264.0 1190.1%
      4096 2 538.8 5681.4 954.4%
      4096 4 528.9 3107.6 487.5%
      4096 8 521.8 2168.9 315.6%
      4096 16 336.5 1256.5 273.4%
      bpc #T PureJava Native % diff
      8192 1 521.1 6842.4 1213.1%
      8192 2 543.0 5510.2 914.8%
      8192 4 526.1 3759.5 614.6%
      8192 8 480.1 2254.1 369.6%
      8192 16 336.4 1213.3 260.6%
      bpc #T PureJava Native % diff
      16384 1 521.7 3695.7 608.3%
      16384 2 540.8 5610.5 937.4%
      16384 4 539.1 3313.0 514.6%
      16384 8 518.7 2198.4 323.8%
      16384 16 337.2 1237.8 267.1%
      bpc #T PureJava Native % diff
      32768 1 532.1 3696.0 594.6%
      32768 2 541.4 4647.4 758.4%
      32768 4 536.6 2325.1 333.3%
      32768 8 518.8 2036.9 292.6%
      32768 16 337.8 1232.0 264.7%
      bpc #T PureJava Native % diff
      65536 1 527.7 8240.0 1461.4%
      65536 2 561.9 5472.3 873.9%
      65536 4 536.5 3725.2 594.4%
      65536 8 482.8 2040.2 322.5%
      65536 16 335.9 1240.4 269.3%

      Elapsed 77.4s

      Attachments

        1. HADOOP-14424.002.patch
          15 kB
          LiXin Ge
        2. HADOOP-14424.patch
          13 kB
          LiXin Ge

        Activity

          People

            GeLiXin LiXin Ge
            GeLiXin LiXin Ge
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: