Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10865

Add a Crc32 chunked verification benchmark for both directly and non-directly buffer cases

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently, it is not easy to compare Crc32 chunked verification implementations. Let's add a benchmark.

      1. c10865_20140717.patch
        19 kB
        Tsz Wo Nicholas Sze
      2. c10865_20160216.patch
        20 kB
        Tsz Wo Nicholas Sze
      3. c10865_20160216b.patch
        20 kB
        Tsz Wo Nicholas Sze
      4. c10865_20160217.patch
        21 kB
        Tsz Wo Nicholas Sze
      5. HADOOP-10865.002.patch
        19 kB
        Tsuyoshi Ozawa

        Issue Links

          Activity

          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          c10865_20140717.patch: copied from HADOOP-10778.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - c10865_20140717.patch: copied from HADOOP-10778 .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
          org.apache.hadoop.ipc.TestIPC
          org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4326//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4326//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.TestSymlinkLocalFSFileContext org.apache.hadoop.ipc.TestIPC org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4326//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4326//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6346/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6346/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1a152c
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6356/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6356/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Refreshed the patch.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Refreshed the patch.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          $ bin/hadoop org.apache.hadoop.util.Crc32PerformanceTest

          bpc #T Zip PureJava % diff Native % diff % diff
          32 1 329.3 685.5 108.1% 1383.0 319.9% 101.7%
          32 2 329.8 685.5 107.8% 1382.4 319.1% 101.7%
          32 4 251.2 627.1 149.7% 1099.4 337.7% 75.3%
          32 8 291.2 494.2 69.7% 1368.9 370.1% 177.0%
          32 16 149.3 290.4 94.5% 814.6 445.5% 180.5%
          bpc #T Zip PureJava % diff Native % diff % diff
          64 1 474.1 816.5 72.2% 1444.6 204.7% 76.9%
          64 2 470.3 817.3 73.8% 1443.9 207.0% 76.7%
          64 4 471.2 816.6 73.3% 1441.2 205.8% 76.5%
          64 8 471.3 813.8 72.7% 1430.1 203.5% 75.7%
          64 16 280.2 423.0 51.0% 959.8 242.6% 126.9%
          bpc #T Zip PureJava % diff Native % diff % diff
          128 1 603.4 900.6 49.3% 1435.7 137.9% 59.4%
          128 2 604.8 901.5 49.1% 1448.7 139.5% 60.7%
          128 4 604.2 900.5 49.0% 1446.9 139.5% 60.7%
          128 8 582.3 895.3 53.7% 1343.9 130.8% 50.1%
          128 16 412.9 493.2 19.4% 1064.6 157.8% 115.9%
          bpc #T Zip PureJava % diff Native % diff % diff
          256 1 701.9 949.1 35.2% 1448.2 106.3% 52.6%
          256 2 702.8 950.9 35.3% 1448.5 106.1% 52.3%
          256 4 697.4 949.4 36.1% 1445.9 107.3% 52.3%
          256 8 682.6 845.0 23.8% 1435.3 110.3% 69.9%
          256 16 528.1 536.5 1.6% 1113.2 110.8% 107.5%
          bpc #T Zip PureJava % diff Native % diff % diff
          512 1 757.1 971.9 28.4% 1437.9 89.9% 47.9%
          512 2 731.0 921.2 26.0% 1437.3 96.6% 56.0%
          512 4 708.1 890.9 25.8% 1435.6 102.7% 61.1%
          512 8 678.6 880.4 29.7% 1424.5 109.9% 61.8%
          512 16 594.3 555.5 -6.5% 1146.5 92.9% 106.4%
          bpc #T Zip PureJava % diff Native % diff % diff
          1024 1 776.0 954.8 23.0% 1444.5 86.1% 51.3%
          1024 2 774.9 943.5 21.8% 1444.3 86.4% 53.1%
          1024 4 772.8 948.1 22.7% 1442.4 86.6% 52.1%
          1024 8 757.5 929.2 22.7% 1431.4 89.0% 54.0%
          1024 16 654.2 581.1 -11.2% 1159.8 77.3% 99.6%
          bpc #T Zip PureJava % diff Native % diff % diff
          2048 1 792.5 955.4 20.6% 1448.6 82.8% 51.6%
          2048 2 793.5 949.7 19.7% 1447.9 82.5% 52.5%
          2048 4 787.1 776.8 -1.3% 1316.2 67.2% 69.4%
          2048 8 762.9 944.1 23.7% 1435.5 88.2% 52.1%
          2048 16 693.9 594.8 -14.3% 1174.0 69.2% 97.4%
          bpc #T Zip PureJava % diff Native % diff % diff
          4096 1 800.7 959.2 19.8% 1450.6 81.2% 51.2%
          4096 2 800.2 958.6 19.8% 1450.3 81.2% 51.3%
          4096 4 801.0 957.3 19.5% 1448.0 80.8% 51.3%
          4096 8 798.0 952.0 19.3% 1437.3 80.1% 51.0%
          4096 16 720.4 605.5 -15.9% 1176.9 63.4% 94.4%
          bpc #T Zip PureJava % diff Native % diff % diff
          8192 1 810.3 966.1 19.2% 1451.3 79.1% 50.2%
          8192 2 808.2 966.0 19.5% 1450.8 79.5% 50.2%
          8192 4 809.4 790.1 -2.4% 1323.2 63.5% 67.5%
          8192 8 807.5 961.0 19.0% 1437.7 78.1% 49.6%
          8192 16 727.3 610.7 -16.0% 1186.4 63.1% 94.3%
          bpc #T Zip PureJava % diff Native % diff % diff
          16384 1 803.9 956.5 19.0% 1451.8 80.6% 51.8%
          16384 2 804.1 957.9 19.1% 1451.3 80.5% 51.5%
          16384 4 803.6 786.8 -2.1% 1449.2 80.3% 84.2%
          16384 8 784.7 953.4 21.5% 1375.1 75.2% 44.2%
          16384 16 730.6 618.4 -15.4% 1181.0 61.7% 91.0%
          bpc #T Zip PureJava % diff Native % diff % diff
          32768 1 807.0 956.8 18.6% 1451.8 79.9% 51.7%
          32768 2 807.6 957.2 18.5% 1451.6 79.8% 51.6%
          32768 4 806.8 958.0 18.7% 1449.7 79.7% 51.3%
          32768 8 803.5 954.0 18.7% 1438.7 79.1% 50.8%
          32768 16 732.4 621.3 -15.2% 1183.8 61.6% 90.5%
          bpc #T Zip PureJava % diff Native % diff % diff
          65536 1 807.4 957.2 18.5% 1452.5 79.9% 51.8%
          65536 2 808.4 958.4 18.6% 1451.6 79.6% 51.5%
          65536 4 807.7 958.5 18.7% 1449.7 79.5% 51.3%
          65536 8 805.5 954.8 18.5% 1439.1 78.7% 50.7%
          65536 16 731.8 620.2 -15.3% 1185.9 62.0% 91.2%
          Show
          ozawa Tsuyoshi Ozawa added a comment - $ bin/hadoop org.apache.hadoop.util.Crc32PerformanceTest bpc #T Zip PureJava % diff Native % diff % diff 32 1 329.3 685.5 108.1% 1383.0 319.9% 101.7% 32 2 329.8 685.5 107.8% 1382.4 319.1% 101.7% 32 4 251.2 627.1 149.7% 1099.4 337.7% 75.3% 32 8 291.2 494.2 69.7% 1368.9 370.1% 177.0% 32 16 149.3 290.4 94.5% 814.6 445.5% 180.5% bpc #T Zip PureJava % diff Native % diff % diff 64 1 474.1 816.5 72.2% 1444.6 204.7% 76.9% 64 2 470.3 817.3 73.8% 1443.9 207.0% 76.7% 64 4 471.2 816.6 73.3% 1441.2 205.8% 76.5% 64 8 471.3 813.8 72.7% 1430.1 203.5% 75.7% 64 16 280.2 423.0 51.0% 959.8 242.6% 126.9% bpc #T Zip PureJava % diff Native % diff % diff 128 1 603.4 900.6 49.3% 1435.7 137.9% 59.4% 128 2 604.8 901.5 49.1% 1448.7 139.5% 60.7% 128 4 604.2 900.5 49.0% 1446.9 139.5% 60.7% 128 8 582.3 895.3 53.7% 1343.9 130.8% 50.1% 128 16 412.9 493.2 19.4% 1064.6 157.8% 115.9% bpc #T Zip PureJava % diff Native % diff % diff 256 1 701.9 949.1 35.2% 1448.2 106.3% 52.6% 256 2 702.8 950.9 35.3% 1448.5 106.1% 52.3% 256 4 697.4 949.4 36.1% 1445.9 107.3% 52.3% 256 8 682.6 845.0 23.8% 1435.3 110.3% 69.9% 256 16 528.1 536.5 1.6% 1113.2 110.8% 107.5% bpc #T Zip PureJava % diff Native % diff % diff 512 1 757.1 971.9 28.4% 1437.9 89.9% 47.9% 512 2 731.0 921.2 26.0% 1437.3 96.6% 56.0% 512 4 708.1 890.9 25.8% 1435.6 102.7% 61.1% 512 8 678.6 880.4 29.7% 1424.5 109.9% 61.8% 512 16 594.3 555.5 -6.5% 1146.5 92.9% 106.4% bpc #T Zip PureJava % diff Native % diff % diff 1024 1 776.0 954.8 23.0% 1444.5 86.1% 51.3% 1024 2 774.9 943.5 21.8% 1444.3 86.4% 53.1% 1024 4 772.8 948.1 22.7% 1442.4 86.6% 52.1% 1024 8 757.5 929.2 22.7% 1431.4 89.0% 54.0% 1024 16 654.2 581.1 -11.2% 1159.8 77.3% 99.6% bpc #T Zip PureJava % diff Native % diff % diff 2048 1 792.5 955.4 20.6% 1448.6 82.8% 51.6% 2048 2 793.5 949.7 19.7% 1447.9 82.5% 52.5% 2048 4 787.1 776.8 -1.3% 1316.2 67.2% 69.4% 2048 8 762.9 944.1 23.7% 1435.5 88.2% 52.1% 2048 16 693.9 594.8 -14.3% 1174.0 69.2% 97.4% bpc #T Zip PureJava % diff Native % diff % diff 4096 1 800.7 959.2 19.8% 1450.6 81.2% 51.2% 4096 2 800.2 958.6 19.8% 1450.3 81.2% 51.3% 4096 4 801.0 957.3 19.5% 1448.0 80.8% 51.3% 4096 8 798.0 952.0 19.3% 1437.3 80.1% 51.0% 4096 16 720.4 605.5 -15.9% 1176.9 63.4% 94.4% bpc #T Zip PureJava % diff Native % diff % diff 8192 1 810.3 966.1 19.2% 1451.3 79.1% 50.2% 8192 2 808.2 966.0 19.5% 1450.8 79.5% 50.2% 8192 4 809.4 790.1 -2.4% 1323.2 63.5% 67.5% 8192 8 807.5 961.0 19.0% 1437.7 78.1% 49.6% 8192 16 727.3 610.7 -16.0% 1186.4 63.1% 94.3% bpc #T Zip PureJava % diff Native % diff % diff 16384 1 803.9 956.5 19.0% 1451.8 80.6% 51.8% 16384 2 804.1 957.9 19.1% 1451.3 80.5% 51.5% 16384 4 803.6 786.8 -2.1% 1449.2 80.3% 84.2% 16384 8 784.7 953.4 21.5% 1375.1 75.2% 44.2% 16384 16 730.6 618.4 -15.4% 1181.0 61.7% 91.0% bpc #T Zip PureJava % diff Native % diff % diff 32768 1 807.0 956.8 18.6% 1451.8 79.9% 51.7% 32768 2 807.6 957.2 18.5% 1451.6 79.8% 51.6% 32768 4 806.8 958.0 18.7% 1449.7 79.7% 51.3% 32768 8 803.5 954.0 18.7% 1438.7 79.1% 50.8% 32768 16 732.4 621.3 -15.2% 1183.8 61.6% 90.5% bpc #T Zip PureJava % diff Native % diff % diff 65536 1 807.4 957.2 18.5% 1452.5 79.9% 51.8% 65536 2 808.4 958.4 18.6% 1451.6 79.6% 51.5% 65536 4 807.7 958.5 18.7% 1449.7 79.5% 51.3% 65536 8 805.5 954.8 18.5% 1439.1 78.7% 50.7% 65536 16 731.8 620.2 -15.3% 1185.9 62.0% 91.2%
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Tsz Wo Nicholas Sze could you take a look at the latest patch?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Tsz Wo Nicholas Sze could you take a look at the latest patch?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 19s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 7s The applied patch generated 1 new checkstyle issues (total was 140, now 136).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 58s Tests passed in hadoop-common.
              61m 47s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12746486/HADOOP-10865.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4025326
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 7s The applied patch generated 1 new checkstyle issues (total was 140, now 136). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 58s Tests passed in hadoop-common.     61m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12746486/HADOOP-10865.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4025326 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The latest patch looks good. Thanks for rebasing it. Since I posted the original patch, let's find another committer to review the patch before committing it.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The latest patch looks good. Thanks for rebasing it. Since I posted the original patch, let's find another committer to review the patch before committing it.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Sure.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Sure.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Colin P. McCabe do you mind taking a look at the patch?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Colin P. McCabe do you mind taking a look at the patch?
          Hide
          iwasakims Masatake Iwasaki added a comment -

          The patch still applies and I agree to add the benchmark. Just a few comments.

          Running org.apache.hadoop.util.TestDataChecksum
          Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.174 sec <<< FAILURE! - in org.apache.hadoop.util.TestDataChecksum
          testCrc32(org.apache.hadoop.util.TestDataChecksum)  Time elapsed: 0.11 sec  <<< FAILURE!
          java.lang.AssertionError: NativeCrc32 is not available
                  at org.apache.hadoop.util.Crc32PerformanceTest.<init>(Crc32PerformanceTest.java:122)
                  at org.apache.hadoop.util.TestDataChecksum.testCrc32(TestDataChecksum.java:203)
          

          The TestDataChecksum#testCrc32 case should be skipped in non-native profile by calling GenericTestUtils#assumeInNativeProfile.

            public void testCrc32() throws Exception {
              new Crc32PerformanceTest(8, 3, true).run();
              new Crc32PerformanceTest(8, 3, false).run();
            }
          

          Do we need the both? Running only Crc32PerformanceTest(8, 3, true) seems to be ok. It makes test time shorter.

          Show
          iwasakims Masatake Iwasaki added a comment - The patch still applies and I agree to add the benchmark. Just a few comments. Running org.apache.hadoop.util.TestDataChecksum Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.174 sec <<< FAILURE! - in org.apache.hadoop.util.TestDataChecksum testCrc32(org.apache.hadoop.util.TestDataChecksum) Time elapsed: 0.11 sec <<< FAILURE! java.lang.AssertionError: NativeCrc32 is not available at org.apache.hadoop.util.Crc32PerformanceTest.<init>(Crc32PerformanceTest.java:122) at org.apache.hadoop.util.TestDataChecksum.testCrc32(TestDataChecksum.java:203) The TestDataChecksum#testCrc32 case should be skipped in non-native profile by calling GenericTestUtils#assumeInNativeProfile . public void testCrc32() throws Exception { new Crc32PerformanceTest(8, 3, true ).run(); new Crc32PerformanceTest(8, 3, false ).run(); } Do we need the both? Running only Crc32PerformanceTest(8, 3, true) seems to be ok. It makes test time shorter.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Crc32PerformanceTest(8, 3, NativeCodeLoader.isNativeCodeLoaded()) would be better. We don't need GenericTestUtils#assumeInNativeProfile in this case.

          Show
          iwasakims Masatake Iwasaki added a comment - Crc32PerformanceTest(8, 3, NativeCodeLoader.isNativeCodeLoaded()) would be better. We don't need GenericTestUtils#assumeInNativeProfile in this case.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 18s trunk passed
          +1 compile 6m 32s trunk passed with JDK v1.8.0_66
          +1 compile 7m 11s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 7s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 44s the patch passed
          +1 compile 6m 27s the patch passed with JDK v1.8.0_66
          +1 javac 6m 27s the patch passed
          +1 compile 7m 11s the patch passed with JDK v1.7.0_91
          +1 javac 7m 11s the patch passed
          -1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 3 new + 132 unchanged - 7 fixed = 135 total (was 139)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          -1 whitespace 0m 0s The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 55s the patch passed
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 8s the patch passed with JDK v1.7.0_91
          +1 unit 8m 17s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 27s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          65m 6s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12746486/HADOOP-10865.002.patch
          JIRA Issue HADOOP-10865
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 55c0f8f9ce19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a7fce9a
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 18s trunk passed +1 compile 6m 32s trunk passed with JDK v1.8.0_66 +1 compile 7m 11s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 44s the patch passed +1 compile 6m 27s the patch passed with JDK v1.8.0_66 +1 javac 6m 27s the patch passed +1 compile 7m 11s the patch passed with JDK v1.7.0_91 +1 javac 7m 11s the patch passed -1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 3 new + 132 unchanged - 7 fixed = 135 total (was 139) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 0s The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 8s the patch passed with JDK v1.7.0_91 +1 unit 8m 17s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 27s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 65m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12746486/HADOOP-10865.002.patch JIRA Issue HADOOP-10865 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 55c0f8f9ce19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a7fce9a Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          c10865_20160216.patch: changes the test to skip native crc but not throwing AssertionError when !NativeCrc32.isAvailable().

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - c10865_20160216.patch: changes the test to skip native crc but not throwing AssertionError when !NativeCrc32.isAvailable().
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 2s trunk passed
          +1 compile 6m 31s trunk passed with JDK v1.8.0_72
          +1 compile 7m 19s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 1m 0s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 44s the patch passed
          +1 compile 6m 47s the patch passed with JDK v1.8.0_72
          +1 javac 6m 47s the patch passed
          +1 compile 7m 34s the patch passed with JDK v1.7.0_95
          +1 javac 7m 34s the patch passed
          -1 checkstyle 0m 26s hadoop-common-project/hadoop-common: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139)
          +1 mvnsite 1m 15s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 11s the patch passed
          +1 javadoc 1m 3s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 10s the patch passed with JDK v1.7.0_95
          +1 unit 9m 34s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 9m 15s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          68m 35s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788139/c10865_20160216.patch
          JIRA Issue HADOOP-10865
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c9a188397cee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4b0e59f
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 2s trunk passed +1 compile 6m 31s trunk passed with JDK v1.8.0_72 +1 compile 7m 19s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 44s the patch passed +1 compile 6m 47s the patch passed with JDK v1.8.0_72 +1 javac 6m 47s the patch passed +1 compile 7m 34s the patch passed with JDK v1.7.0_95 +1 javac 7m 34s the patch passed -1 checkstyle 0m 26s hadoop-common-project/hadoop-common: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139) +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 11s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_95 +1 unit 9m 34s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 9m 15s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 68m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788139/c10865_20160216.patch JIRA Issue HADOOP-10865 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c9a188397cee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4b0e59f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Thanks for the update, Tsz Wo Nicholas Sze. TestDataChecksum#testBulkOps fails in non-native profile. There seems to be a problem in calculation of errPos in DataChecksum#verifyChunked.

          java.lang.AssertionError: expected:<512> but was:<511>
                  at org.junit.Assert.fail(Assert.java:88)
                  at org.junit.Assert.failNotEquals(Assert.java:743)
                  at org.junit.Assert.assertEquals(Assert.java:118)
                  at org.junit.Assert.assertEquals(Assert.java:555)
                  at org.junit.Assert.assertEquals(Assert.java:542)
                  at org.apache.hadoop.util.TestDataChecksum$Harness.testCorrectness(TestDataChecksum.java:120)
                  at org.apache.hadoop.util.TestDataChecksum.doBulkTest(TestDataChecksum.java:132)
                  at org.apache.hadoop.util.TestDataChecksum.testBulkOps(TestDataChecksum.java:51)
          
          Show
          iwasakims Masatake Iwasaki added a comment - Thanks for the update, Tsz Wo Nicholas Sze . TestDataChecksum#testBulkOps fails in non-native profile. There seems to be a problem in calculation of errPos in DataChecksum#verifyChunked . java.lang.AssertionError: expected:<512> but was:<511> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.util.TestDataChecksum$Harness.testCorrectness(TestDataChecksum.java:120) at org.apache.hadoop.util.TestDataChecksum.doBulkTest(TestDataChecksum.java:132) at org.apache.hadoop.util.TestDataChecksum.testBulkOps(TestDataChecksum.java:51)
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Good catch! The errPos was wrong.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Good catch! The errPos was wrong.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          c10865_20160216b.patch: fixes the bug.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - c10865_20160216b.patch: fixes the bug.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 2s trunk passed
          +1 compile 7m 7s trunk passed with JDK v1.8.0_72
          +1 compile 7m 21s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 58s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 43s the patch passed
          +1 compile 6m 52s the patch passed with JDK v1.8.0_72
          +1 javac 6m 52s the patch passed
          +1 compile 7m 31s the patch passed with JDK v1.7.0_95
          +1 javac 7m 31s the patch passed
          -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139)
          +1 mvnsite 1m 7s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 10s the patch passed with JDK v1.7.0_95
          +1 unit 8m 24s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 8m 21s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          66m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788159/c10865_20160216b.patch
          JIRA Issue HADOOP-10865
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 73a5f2ee29e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 77ba5ad
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 2s trunk passed +1 compile 7m 7s trunk passed with JDK v1.8.0_72 +1 compile 7m 21s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 58s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 43s the patch passed +1 compile 6m 52s the patch passed with JDK v1.8.0_72 +1 javac 6m 52s the patch passed +1 compile 7m 31s the patch passed with JDK v1.7.0_95 +1 javac 7m 31s the patch passed -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139) +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_95 +1 unit 8m 24s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 8m 21s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 66m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788159/c10865_20160216b.patch JIRA Issue HADOOP-10865 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 73a5f2ee29e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 77ba5ad Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Hmm.. TestDFSClientRetries#testRetryOnChecksumFailure expects that the description of ChecksumException contains "Checksum error". (Jenkins did not run the test because it is under hadoop-hdfs.)

          java.lang.AssertionError: Expected to find 'Checksum error' but got unexpected exception:org.apache.hadoop.fs.ChecksumException: Checksum CRC32C not matched for file /corrupted at position 0: expected=3047312A but computed=B5E7A6EF, algorithm=PureJavaCrc32C
                  ...
                  at org.apache.hadoop.hdfs.TestDFSClientRetries.testRetryOnChecksumFailure(TestDFSClientRetries.java:930)
          

          The assertion or DataChecksum#throwChecksumException should be fixed to match these.

                try {
                    dis.read(arr, 0, (int)FILE_LENGTH);
                    fail("Expected ChecksumException not thrown");
                  } catch (Exception ex) {
                    GenericTestUtils.assertExceptionContains(
                        "Checksum error", ex);
                  }
                }
          
          Show
          iwasakims Masatake Iwasaki added a comment - Hmm.. TestDFSClientRetries#testRetryOnChecksumFailure expects that the description of ChecksumException contains "Checksum error". (Jenkins did not run the test because it is under hadoop-hdfs.) java.lang.AssertionError: Expected to find 'Checksum error' but got unexpected exception:org.apache.hadoop.fs.ChecksumException: Checksum CRC32C not matched for file /corrupted at position 0: expected=3047312A but computed=B5E7A6EF, algorithm=PureJavaCrc32C ... at org.apache.hadoop.hdfs.TestDFSClientRetries.testRetryOnChecksumFailure(TestDFSClientRetries.java:930) The assertion or DataChecksum#throwChecksumException should be fixed to match these. try { dis.read(arr, 0, ( int )FILE_LENGTH); fail( "Expected ChecksumException not thrown" ); } catch (Exception ex) { GenericTestUtils.assertExceptionContains( "Checksum error" , ex); } }
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks. Here is a patch fixing TestDFSClientRetries.

          c10865_20160217.patch

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks. Here is a patch fixing TestDFSClientRetries. c10865_20160217.patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 34s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 6m 55s trunk passed with JDK v1.8.0_72
          +1 compile 6m 50s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 1m 56s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 29s trunk passed
          +1 javadoc 2m 4s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 0s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 6m 17s the patch passed with JDK v1.8.0_72
          +1 javac 6m 17s the patch passed
          +1 compile 6m 46s the patch passed with JDK v1.7.0_95
          +1 javac 6m 46s the patch passed
          -1 checkstyle 1m 5s root: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139)
          +1 mvnsite 1m 55s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 57s the patch passed
          +1 javadoc 2m 12s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95
          +1 unit 7m 13s hadoop-common in the patch passed with JDK v1.8.0_72.
          -1 unit 56m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 53m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 30s Patch does not generate ASF License warnings.
          188m 1s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileAppend
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788306/c10865_20160217.patch
          JIRA Issue HADOOP-10865
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1ea2a9a7809e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 76fab26
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 34s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 6m 55s trunk passed with JDK v1.8.0_72 +1 compile 6m 50s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 56s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 29s trunk passed +1 javadoc 2m 4s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 0s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 6m 17s the patch passed with JDK v1.8.0_72 +1 javac 6m 17s the patch passed +1 compile 6m 46s the patch passed with JDK v1.7.0_95 +1 javac 6m 46s the patch passed -1 checkstyle 1m 5s root: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139) +1 mvnsite 1m 55s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 57s the patch passed +1 javadoc 2m 12s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 51s the patch passed with JDK v1.7.0_95 +1 unit 7m 13s hadoop-common in the patch passed with JDK v1.8.0_72. -1 unit 56m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 53m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 30s Patch does not generate ASF License warnings. 188m 1s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileAppend JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788306/c10865_20160217.patch JIRA Issue HADOOP-10865 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1ea2a9a7809e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 76fab26 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8649/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          +1. The failure of TestFileAppend and TestShortCircuitCache is not related to the patch.

          Show
          iwasakims Masatake Iwasaki added a comment - +1. The failure of TestFileAppend and TestShortCircuitCache is not related to the patch.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Masatake Iwasaki for reviewing and testing the patches!

          I have committed this.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Masatake Iwasaki for reviewing and testing the patches! I have committed this.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9324 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9324/)
          HADOOP-10865. Add a Crc32 chunked verification benchmark for both (szetszwo: rev bbfaf3c2712c9ba82b0f8423bdeb314bf505a692)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDataChecksum.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/Crc32PerformanceTest.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9324 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9324/ ) HADOOP-10865 . Add a Crc32 chunked verification benchmark for both (szetszwo: rev bbfaf3c2712c9ba82b0f8423bdeb314bf505a692) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDataChecksum.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/Crc32PerformanceTest.java

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              szetszwo Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development