Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: common
    • Labels:

      Description

      The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new test to compare Crc32C chunked verification implementations.
      This test is based on Crc32PerformanceTest, what I have done in this test is:
      1.CRC32C performance test.
      2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it from this test.
      3.User can choose either direct buffer or non-directly buffer to run this test manually.
      4.Take use of verifyChunkedSumsByteArray for native to support non-directly native test.

      The test result in my environment is:
      [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
      java.version = 1.8.0_111
      java.runtime.name = Java(TM) SE Runtime Environment
      java.runtime.version = 1.8.0_111-b14
      java.vm.version = 25.111-b14
      java.vm.vendor = Oracle Corporation
      java.vm.name = Java HotSpot(TM) 64-Bit Server VM
      java.vm.specification.version = 1.8
      java.specification.version = 1.8
      os.arch = amd64
      os.name = Linux
      os.version = 2.6.33.20
      Data Length = 64 MB
      Trials = 5

      Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)

      bpc #T PureJava Native % diff
      32 1 394.0 4156.2 954.9%
      32 2 400.5 3679.7 818.7%
      32 4 401.8 2657.0 561.3%
      32 8 389.1 1633.8 319.9%
      32 16 222.2 1116.3 402.5%
      bpc #T PureJava Native % diff
      64 1 465.0 5931.0 1175.5%
      64 2 468.8 1839.2 292.3%
      64 4 460.4 2968.3 544.7%
      64 8 452.4 1925.7 325.6%
      64 16 246.9 1291.8 423.3%
      bpc #T PureJava Native % diff
      128 1 522.0 6147.8 1077.6%
      128 2 366.0 4758.5 1200.2%
      128 4 307.8 3265.1 960.8%
      128 8 283.6 2092.2 637.6%
      128 16 219.9 1226.1 457.6%
      bpc #T PureJava Native % diff
      256 1 550.7 3177.6 477.0%
      256 2 538.6 1933.2 258.9%
      256 4 427.2 3278.1 667.3%
      256 8 420.8 2272.3 440.0%
      256 16 294.0 1311.2 346.0%
      bpc #T PureJava Native % diff
      512 1 553.4 3690.4 566.9%
      512 2 455.6 4974.1 991.7%
      512 4 494.2 3406.4 589.2%
      512 8 431.4 2257.0 423.2%
      512 16 316.3 1272.0 302.2%
      bpc #T PureJava Native % diff
      1024 1 566.1 3520.0 521.8%
      1024 2 508.7 4437.4 772.3%
      1024 4 520.7 3422.6 557.4%
      1024 8 501.8 2124.7 323.4%
      1024 16 340.6 1305.0 283.2%
      bpc #T PureJava Native % diff
      2048 1 535.1 5438.5 916.4%
      2048 2 537.3 4668.3 768.8%
      2048 4 529.2 2417.2 356.7%
      2048 8 485.1 2249.8 363.8%
      2048 16 334.3 1265.7 278.6%
      bpc #T PureJava Native % diff
      4096 1 563.0 7264.0 1190.1%
      4096 2 538.8 5681.4 954.4%
      4096 4 528.9 3107.6 487.5%
      4096 8 521.8 2168.9 315.6%
      4096 16 336.5 1256.5 273.4%
      bpc #T PureJava Native % diff
      8192 1 521.1 6842.4 1213.1%
      8192 2 543.0 5510.2 914.8%
      8192 4 526.1 3759.5 614.6%
      8192 8 480.1 2254.1 369.6%
      8192 16 336.4 1213.3 260.6%
      bpc #T PureJava Native % diff
      16384 1 521.7 3695.7 608.3%
      16384 2 540.8 5610.5 937.4%
      16384 4 539.1 3313.0 514.6%
      16384 8 518.7 2198.4 323.8%
      16384 16 337.2 1237.8 267.1%
      bpc #T PureJava Native % diff
      32768 1 532.1 3696.0 594.6%
      32768 2 541.4 4647.4 758.4%
      32768 4 536.6 2325.1 333.3%
      32768 8 518.8 2036.9 292.6%
      32768 16 337.8 1232.0 264.7%
      bpc #T PureJava Native % diff
      65536 1 527.7 8240.0 1461.4%
      65536 2 561.9 5472.3 873.9%
      65536 4 536.5 3725.2 594.4%
      65536 8 482.8 2040.2 322.5%
      65536 16 335.9 1240.4 269.3%

      Elapsed 77.4s

      1. HADOOP-14424.002.patch
        15 kB
        LiXin Ge
      2. HADOOP-14424.patch
        13 kB
        LiXin Ge

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 32s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 12m 17s trunk passed
        +1 compile 12m 25s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        -1 findbugs 1m 20s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        +1 javadoc 0m 42s trunk passed
        +1 mvninstall 0m 40s the patch passed
        +1 compile 12m 1s the patch passed
        +1 javac 12m 1s the patch passed
        -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 21 new + 13 unchanged - 0 fixed = 34 total (was 13)
        +1 mvnsite 1m 1s the patch passed
        +1 mvneclipse 0m 18s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 22s the patch passed
        +1 javadoc 0m 48s the patch passed
        -1 unit 6m 54s hadoop-common in the patch failed.
        +1 asflicense 0m 28s The patch does not generate ASF License warnings.
        54m 49s



        Reason Tests
        Failed junit tests hadoop.ha.TestZKFailoverControllerStress



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14424
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868216/HADOOP-14424.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 19a3ed2cd857 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c48f297
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 12m 17s trunk passed +1 compile 12m 25s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 1m 20s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 40s the patch passed +1 compile 12m 1s the patch passed +1 javac 12m 1s the patch passed -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 21 new + 13 unchanged - 0 fixed = 34 total (was 13) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 22s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 6m 54s hadoop-common in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 54m 49s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverControllerStress Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14424 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868216/HADOOP-14424.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 19a3ed2cd857 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c48f297 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12323/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        GeLiXin LiXin Ge added a comment -

        update the patch against the checkstyle warnings

        Show
        GeLiXin LiXin Ge added a comment - update the patch against the checkstyle warnings
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 15m 30s trunk passed
        +1 compile 13m 16s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 1m 5s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        -1 findbugs 1m 27s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        +1 javadoc 0m 50s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 14m 13s the patch passed
        +1 javac 14m 13s the patch passed
        -0 checkstyle 0m 37s hadoop-common-project/hadoop-common: The patch generated 12 new + 13 unchanged - 0 fixed = 25 total (was 13)
        +1 mvnsite 1m 11s the patch passed
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 51s the patch passed
        +1 unit 7m 45s hadoop-common in the patch passed.
        +1 asflicense 0m 36s The patch does not generate ASF License warnings.
        63m 17s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14424
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868240/HADOOP-14424.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 88db4147e228 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c48f297
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 15m 30s trunk passed +1 compile 13m 16s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 20s trunk passed -1 findbugs 1m 27s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 14m 13s the patch passed +1 javac 14m 13s the patch passed -0 checkstyle 0m 37s hadoop-common-project/hadoop-common: The patch generated 12 new + 13 unchanged - 0 fixed = 25 total (was 13) +1 mvnsite 1m 11s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 51s the patch passed +1 unit 7m 45s hadoop-common in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 63m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14424 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868240/HADOOP-14424.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 88db4147e228 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c48f297 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12326/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        GeLiXin LiXin Ge added a comment -

        try to fix [VisibilityModifier] style warnings. not very sure....

        Show
        GeLiXin LiXin Ge added a comment - try to fix [VisibilityModifier] style warnings. not very sure....
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 12m 8s trunk passed
        +1 compile 12m 24s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 22s trunk passed
        -1 findbugs 1m 18s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        +1 javadoc 0m 48s trunk passed
        +1 mvninstall 0m 34s the patch passed
        +1 compile 12m 0s the patch passed
        +1 javac 12m 0s the patch passed
        -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 7 new + 13 unchanged - 0 fixed = 20 total (was 13)
        +1 mvnsite 0m 58s the patch passed
        +1 mvneclipse 0m 23s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 30s the patch passed
        +1 javadoc 0m 48s the patch passed
        +1 unit 7m 26s hadoop-common in the patch passed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        55m 27s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14424
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868301/HADOOP-14424.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ee463c09e5f0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c48f297
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 12m 8s trunk passed +1 compile 12m 24s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 22s trunk passed -1 findbugs 1m 18s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 34s the patch passed +1 compile 12m 0s the patch passed +1 javac 12m 0s the patch passed -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 7 new + 13 unchanged - 0 fixed = 20 total (was 13) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 7m 26s hadoop-common in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 55m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14424 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868301/HADOOP-14424.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ee463c09e5f0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c48f297 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12333/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        GeLiXin LiXin Ge added a comment -

        The checkstyle warnings of "Variable * must be private and have accessor methods. [VisibilityModifier]" seems too sensitive in test code.
        And there were so many such type warnings in the existing code, is it really worthy fixing?

        Show
        GeLiXin LiXin Ge added a comment - The checkstyle warnings of "Variable * must be private and have accessor methods. [VisibilityModifier] " seems too sensitive in test code. And there were so many such type warnings in the existing code, is it really worthy fixing?
        Hide
        GeLiXin LiXin Ge added a comment -

        Hi, Masatake Iwasaki, sorry to disturb you, I'm wondering what's the situation of this patch now?
        This CRC32C patch is based on your CRC32 patch, what shall I do to carry the work a step forward?
        Thanks!

        Show
        GeLiXin LiXin Ge added a comment - Hi, Masatake Iwasaki , sorry to disturb you, I'm wondering what's the situation of this patch now? This CRC32C patch is based on your CRC32 patch, what shall I do to carry the work a step forward? Thanks!
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks for working on this, LiXin Ge.

        It would be nice if duplicate code in Crc32PerformanceTest and Crc32CPerformanceTest. I think it is ok to just add test cases for CRC32C to existing Crc32PerformanceTest. In that case, displaying diffs to all of previous benchmark results could be noisy. It should be omitted or just print diff to one of the results.

        //compare result with previous
        for(BenchResult p : previous) {
                  final double diff = (result.mbps - p.mbps) / p.mbps * 100;
                  printCell(String.format("%5.1f%%", diff), diffStr.length(), out);
                }
                previous.add(result);
              }
        
        Show
        iwasakims Masatake Iwasaki added a comment - Thanks for working on this, LiXin Ge . It would be nice if duplicate code in Crc32PerformanceTest and Crc32CPerformanceTest. I think it is ok to just add test cases for CRC32C to existing Crc32PerformanceTest. In that case, displaying diffs to all of previous benchmark results could be noisy. It should be omitted or just print diff to one of the results. //compare result with previous for(BenchResult p : previous) { final double diff = (result.mbps - p.mbps) / p.mbps * 100; printCell(String.format("%5.1f%%", diff), diffStr.length(), out); } previous.add(result); }
        Hide
        GeLiXin LiXin Ge added a comment -

        Thanks for your advice, Masatake Iwasaki.
        I'm done to combine the two performance functions into one test, some interface and functions was modified to support both Crc32 and Crc32C.
        It's only print diff to one of the results now and a part of the output of Crc32PerformanceTest is:
        Direct Buffer():

        ...

        bpc #T Zip PureJava % diff PureJavaC % diff Native % diff NativeC % diff
        32 1 154.5 357.2 131.2% 349.2 -2.2% 783.3 124.3% 3146.0 301.6%
        32 2 154.3 354.0 129.4% 350.2 -1.1% 794.8 126.9% 1710.1 115.2%
        32 4 98.4 294.0 198.8% 339.6 15.5% 782.2 130.3% 2615.0 234.3%
        32 8 112.1 231.7 106.6% 199.8 -13.7% 708.4 254.5% 1833.0 158.7%
        32 16 65.0 137.2 111.2% 144.2 5.1% 442.9 207.1% 1107.7 150.1%

        ...

        Non-direct Buffer:

        ...

        bpc #T Zip PureJava % diff PureJavaC % diff NativeC % diff
        32 1 161.3 402.0 149.3% 396.9 -1.3% 3028.9 663.1%
        32 2 158.3 403.1 154.6% 396.4 -1.7% 1697.1 328.1%
        32 4 156.8 402.7 156.8% 394.7 -2.0% 1898.1 380.9%
        32 8 157.4 402.8 155.9% 392.7 -2.5% 1897.4 383.2%
        32 16 91.4 224.1 145.2% 219.9 -1.9% 1202.8 446.9%

        ...

        Show
        GeLiXin LiXin Ge added a comment - Thanks for your advice, Masatake Iwasaki . I'm done to combine the two performance functions into one test, some interface and functions was modified to support both Crc32 and Crc32C. It's only print diff to one of the results now and a part of the output of Crc32PerformanceTest is: Direct Buffer(): ... bpc #T Zip PureJava % diff PureJavaC % diff Native % diff NativeC % diff 32 1 154.5 357.2 131.2% 349.2 -2.2% 783.3 124.3% 3146.0 301.6% 32 2 154.3 354.0 129.4% 350.2 -1.1% 794.8 126.9% 1710.1 115.2% 32 4 98.4 294.0 198.8% 339.6 15.5% 782.2 130.3% 2615.0 234.3% 32 8 112.1 231.7 106.6% 199.8 -13.7% 708.4 254.5% 1833.0 158.7% 32 16 65.0 137.2 111.2% 144.2 5.1% 442.9 207.1% 1107.7 150.1% ... Non-direct Buffer: ... bpc #T Zip PureJava % diff PureJavaC % diff NativeC % diff 32 1 161.3 402.0 149.3% 396.9 -1.3% 3028.9 663.1% 32 2 158.3 403.1 154.6% 396.4 -1.7% 1697.1 328.1% 32 4 156.8 402.7 156.8% 394.7 -2.0% 1898.1 380.9% 32 8 157.4 402.8 155.9% 392.7 -2.5% 1897.4 383.2% 32 16 91.4 224.1 145.2% 219.9 -1.9% 1202.8 446.9% ...
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 19s trunk passed
        +1 compile 13m 42s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 1m 4s trunk passed
        -1 findbugs 1m 23s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        +1 javadoc 0m 50s trunk passed
        +1 mvninstall 0m 39s the patch passed
        +1 compile 10m 18s the patch passed
        +1 javac 10m 18s the patch passed
        +1 checkstyle 0m 36s hadoop-common-project/hadoop-common: The patch generated 0 new + 7 unchanged - 12 fixed = 7 total (was 19)
        +1 mvnsite 1m 2s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 36s the patch passed
        +1 javadoc 0m 50s the patch passed
        -1 unit 8m 3s hadoop-common in the patch failed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        56m 37s



        Reason Tests
        Failed junit tests hadoop.test.TestLambdaTestUtils



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14424
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872952/HADOOP-14424.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 40633afbc2db 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6ed54f3
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 19s trunk passed +1 compile 13m 42s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 4s trunk passed -1 findbugs 1m 23s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 10m 18s the patch passed +1 javac 10m 18s the patch passed +1 checkstyle 0m 36s hadoop-common-project/hadoop-common: The patch generated 0 new + 7 unchanged - 12 fixed = 7 total (was 19) +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 50s the patch passed -1 unit 8m 3s hadoop-common in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 56m 37s Reason Tests Failed junit tests hadoop.test.TestLambdaTestUtils Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14424 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872952/HADOOP-14424.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 40633afbc2db 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ed54f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12530/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        +1, committing this.

        Show
        iwasakims Masatake Iwasaki added a comment - +1, committing this.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Committed to branch-2 and trunk. Thanks, LiXin Ge.

        Show
        iwasakims Masatake Iwasaki added a comment - Committed to branch-2 and trunk. Thanks, LiXin Ge .
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11882 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11882/)
        HADOOP-14424. Add CRC32C performance test. Contributed by LiXin Ge. (iwasakims: rev 75043d342935828cfe8bdabfab8784c83f797887)

        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/Crc32PerformanceTest.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11882 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11882/ ) HADOOP-14424 . Add CRC32C performance test. Contributed by LiXin Ge. (iwasakims: rev 75043d342935828cfe8bdabfab8784c83f797887) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/Crc32PerformanceTest.java
        Hide
        GeLiXin LiXin Ge added a comment -

        That's my pleasure, Masatake Iwasaki. Thanks for your advice and committing this.

        Show
        GeLiXin LiXin Ge added a comment - That's my pleasure, Masatake Iwasaki . Thanks for your advice and committing this.

          People

          • Assignee:
            GeLiXin LiXin Ge
            Reporter:
            GeLiXin LiXin Ge
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development