Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: util
-
Labels:
-
Hadoop Flags:Reviewed
Description
Currently, it is not easy to compare Crc32 chunked verification implementations. Let's add a benchmark.
Issue Links
- relates to
-
HADOOP-10778
Use NativeCrc32 only if it is faster
-
- Patch Available
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.fs.TestSymlinkLocalFSFileContext
org.apache.hadoop.ipc.TestIPC
org.apache.hadoop.fs.TestSymlinkLocalFSFileSystem
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4326//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4326//console
This message is automatically generated.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | patch | 0m 0s | The patch command could not apply the patch during dryrun. |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6346/console |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | patch | 0m 0s | The patch command could not apply the patch during dryrun. |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12656753/c10865_20140717.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6356/console |
This message was automatically generated.
Refreshed the patch.
$ bin/hadoop org.apache.hadoop.util.Crc32PerformanceTest
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
|---|---|---|---|---|---|---|---|
| 32 | 1 | 329.3 | 685.5 | 108.1% | 1383.0 | 319.9% | 101.7% |
| 32 | 2 | 329.8 | 685.5 | 107.8% | 1382.4 | 319.1% | 101.7% |
| 32 | 4 | 251.2 | 627.1 | 149.7% | 1099.4 | 337.7% | 75.3% |
| 32 | 8 | 291.2 | 494.2 | 69.7% | 1368.9 | 370.1% | 177.0% |
| 32 | 16 | 149.3 | 290.4 | 94.5% | 814.6 | 445.5% | 180.5% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 64 | 1 | 474.1 | 816.5 | 72.2% | 1444.6 | 204.7% | 76.9% |
| 64 | 2 | 470.3 | 817.3 | 73.8% | 1443.9 | 207.0% | 76.7% |
| 64 | 4 | 471.2 | 816.6 | 73.3% | 1441.2 | 205.8% | 76.5% |
| 64 | 8 | 471.3 | 813.8 | 72.7% | 1430.1 | 203.5% | 75.7% |
| 64 | 16 | 280.2 | 423.0 | 51.0% | 959.8 | 242.6% | 126.9% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 128 | 1 | 603.4 | 900.6 | 49.3% | 1435.7 | 137.9% | 59.4% |
| 128 | 2 | 604.8 | 901.5 | 49.1% | 1448.7 | 139.5% | 60.7% |
| 128 | 4 | 604.2 | 900.5 | 49.0% | 1446.9 | 139.5% | 60.7% |
| 128 | 8 | 582.3 | 895.3 | 53.7% | 1343.9 | 130.8% | 50.1% |
| 128 | 16 | 412.9 | 493.2 | 19.4% | 1064.6 | 157.8% | 115.9% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 256 | 1 | 701.9 | 949.1 | 35.2% | 1448.2 | 106.3% | 52.6% |
| 256 | 2 | 702.8 | 950.9 | 35.3% | 1448.5 | 106.1% | 52.3% |
| 256 | 4 | 697.4 | 949.4 | 36.1% | 1445.9 | 107.3% | 52.3% |
| 256 | 8 | 682.6 | 845.0 | 23.8% | 1435.3 | 110.3% | 69.9% |
| 256 | 16 | 528.1 | 536.5 | 1.6% | 1113.2 | 110.8% | 107.5% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 512 | 1 | 757.1 | 971.9 | 28.4% | 1437.9 | 89.9% | 47.9% |
| 512 | 2 | 731.0 | 921.2 | 26.0% | 1437.3 | 96.6% | 56.0% |
| 512 | 4 | 708.1 | 890.9 | 25.8% | 1435.6 | 102.7% | 61.1% |
| 512 | 8 | 678.6 | 880.4 | 29.7% | 1424.5 | 109.9% | 61.8% |
| 512 | 16 | 594.3 | 555.5 | -6.5% | 1146.5 | 92.9% | 106.4% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 1024 | 1 | 776.0 | 954.8 | 23.0% | 1444.5 | 86.1% | 51.3% |
| 1024 | 2 | 774.9 | 943.5 | 21.8% | 1444.3 | 86.4% | 53.1% |
| 1024 | 4 | 772.8 | 948.1 | 22.7% | 1442.4 | 86.6% | 52.1% |
| 1024 | 8 | 757.5 | 929.2 | 22.7% | 1431.4 | 89.0% | 54.0% |
| 1024 | 16 | 654.2 | 581.1 | -11.2% | 1159.8 | 77.3% | 99.6% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 2048 | 1 | 792.5 | 955.4 | 20.6% | 1448.6 | 82.8% | 51.6% |
| 2048 | 2 | 793.5 | 949.7 | 19.7% | 1447.9 | 82.5% | 52.5% |
| 2048 | 4 | 787.1 | 776.8 | -1.3% | 1316.2 | 67.2% | 69.4% |
| 2048 | 8 | 762.9 | 944.1 | 23.7% | 1435.5 | 88.2% | 52.1% |
| 2048 | 16 | 693.9 | 594.8 | -14.3% | 1174.0 | 69.2% | 97.4% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 4096 | 1 | 800.7 | 959.2 | 19.8% | 1450.6 | 81.2% | 51.2% |
| 4096 | 2 | 800.2 | 958.6 | 19.8% | 1450.3 | 81.2% | 51.3% |
| 4096 | 4 | 801.0 | 957.3 | 19.5% | 1448.0 | 80.8% | 51.3% |
| 4096 | 8 | 798.0 | 952.0 | 19.3% | 1437.3 | 80.1% | 51.0% |
| 4096 | 16 | 720.4 | 605.5 | -15.9% | 1176.9 | 63.4% | 94.4% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 8192 | 1 | 810.3 | 966.1 | 19.2% | 1451.3 | 79.1% | 50.2% |
| 8192 | 2 | 808.2 | 966.0 | 19.5% | 1450.8 | 79.5% | 50.2% |
| 8192 | 4 | 809.4 | 790.1 | -2.4% | 1323.2 | 63.5% | 67.5% |
| 8192 | 8 | 807.5 | 961.0 | 19.0% | 1437.7 | 78.1% | 49.6% |
| 8192 | 16 | 727.3 | 610.7 | -16.0% | 1186.4 | 63.1% | 94.3% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 16384 | 1 | 803.9 | 956.5 | 19.0% | 1451.8 | 80.6% | 51.8% |
| 16384 | 2 | 804.1 | 957.9 | 19.1% | 1451.3 | 80.5% | 51.5% |
| 16384 | 4 | 803.6 | 786.8 | -2.1% | 1449.2 | 80.3% | 84.2% |
| 16384 | 8 | 784.7 | 953.4 | 21.5% | 1375.1 | 75.2% | 44.2% |
| 16384 | 16 | 730.6 | 618.4 | -15.4% | 1181.0 | 61.7% | 91.0% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 32768 | 1 | 807.0 | 956.8 | 18.6% | 1451.8 | 79.9% | 51.7% |
| 32768 | 2 | 807.6 | 957.2 | 18.5% | 1451.6 | 79.8% | 51.6% |
| 32768 | 4 | 806.8 | 958.0 | 18.7% | 1449.7 | 79.7% | 51.3% |
| 32768 | 8 | 803.5 | 954.0 | 18.7% | 1438.7 | 79.1% | 50.8% |
| 32768 | 16 | 732.4 | 621.3 | -15.2% | 1183.8 | 61.6% | 90.5% |
| bpc | #T | Zip | PureJava | % diff | Native | % diff | % diff |
| 65536 | 1 | 807.4 | 957.2 | 18.5% | 1452.5 | 79.9% | 51.8% |
| 65536 | 2 | 808.4 | 958.4 | 18.6% | 1451.6 | 79.6% | 51.5% |
| 65536 | 4 | 807.7 | 958.5 | 18.7% | 1449.7 | 79.5% | 51.3% |
| 65536 | 8 | 805.5 | 954.8 | 18.5% | 1439.1 | 78.7% | 50.7% |
| 65536 | 16 | 731.8 | 620.2 | -15.3% | 1185.9 | 62.0% | 91.2% |
Tsz Wo Nicholas Sze could you take a look at the latest patch?
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 16m 19s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | javac | 7m 33s | There were no new javac warning messages. |
| +1 | javadoc | 9m 39s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 7s | The applied patch generated 1 new checkstyle issues (total was 140, now 136). |
| -1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | install | 1m 20s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 1m 51s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | common tests | 22m 58s | Tests passed in hadoop-common. |
| 61m 47s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12746486/HADOOP-10865.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4025326 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/diffcheckstylehadoop-common.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/whitespace.txt |
| hadoop-common test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/artifact/patchprocess/testrun_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/7322/console |
This message was automatically generated.
The latest patch looks good. Thanks for rebasing it. Since I posted the original patch, let's find another committer to review the patch before committing it.
Sure.
Colin P. McCabe do you mind taking a look at the patch?
The patch still applies and I agree to add the benchmark. Just a few comments.
Running org.apache.hadoop.util.TestDataChecksum
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.174 sec <<< FAILURE! - in org.apache.hadoop.util.TestDataChecksum
testCrc32(org.apache.hadoop.util.TestDataChecksum) Time elapsed: 0.11 sec <<< FAILURE!
java.lang.AssertionError: NativeCrc32 is not available
at org.apache.hadoop.util.Crc32PerformanceTest.<init>(Crc32PerformanceTest.java:122)
at org.apache.hadoop.util.TestDataChecksum.testCrc32(TestDataChecksum.java:203)
The TestDataChecksum#testCrc32 case should be skipped in non-native profile by calling GenericTestUtils#assumeInNativeProfile.
public void testCrc32() throws Exception { new Crc32PerformanceTest(8, 3, true).run(); new Crc32PerformanceTest(8, 3, false).run(); }
Do we need the both? Running only Crc32PerformanceTest(8, 3, true) seems to be ok. It makes test time shorter.
Crc32PerformanceTest(8, 3, NativeCodeLoader.isNativeCodeLoaded()) would be better. We don't need GenericTestUtils#assumeInNativeProfile in this case.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 11s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 7m 18s | trunk passed |
| +1 | compile | 6m 32s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 7m 11s | trunk passed with JDK v1.7.0_91 |
| +1 | checkstyle | 0m 24s | trunk passed |
| +1 | mvnsite | 1m 6s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 1m 39s | trunk passed |
| +1 | javadoc | 0m 57s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 7s | trunk passed with JDK v1.7.0_91 |
| +1 | mvninstall | 0m 44s | the patch passed |
| +1 | compile | 6m 27s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 6m 27s | the patch passed |
| +1 | compile | 7m 11s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 7m 11s | the patch passed |
| -1 | checkstyle | 0m 22s | hadoop-common-project/hadoop-common: patch generated 3 new + 132 unchanged - 7 fixed = 135 total (was 139) |
| +1 | mvnsite | 1m 5s | the patch passed |
| +1 | mvneclipse | 0m 14s | the patch passed |
| -1 | whitespace | 0m 0s | The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | findbugs | 1m 55s | the patch passed |
| +1 | javadoc | 0m 57s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 8s | the patch passed with JDK v1.7.0_91 |
| +1 | unit | 8m 17s | hadoop-common in the patch passed with JDK v1.8.0_66. |
| +1 | unit | 8m 27s | hadoop-common in the patch passed with JDK v1.7.0_91. |
| +1 | asflicense | 0m 24s | Patch does not generate ASF License warnings. |
| 65m 6s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12746486/HADOOP-10865.002.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 55c0f8f9ce19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a7fce9a |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/artifact/patchprocess/whitespace-eol.txt |
| JDK v1.7.0_91 Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Max memory used | 77MB |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8583/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
c10865_20160216.patch: changes the test to skip native crc but not throwing AssertionError when !NativeCrc32.isAvailable().
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 11s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 7m 2s | trunk passed |
| +1 | compile | 6m 31s | trunk passed with JDK v1.8.0_72 |
| +1 | compile | 7m 19s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 23s | trunk passed |
| +1 | mvnsite | 1m 7s | trunk passed |
| +1 | mvneclipse | 0m 14s | trunk passed |
| +1 | findbugs | 1m 43s | trunk passed |
| +1 | javadoc | 1m 0s | trunk passed with JDK v1.8.0_72 |
| +1 | javadoc | 1m 7s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 44s | the patch passed |
| +1 | compile | 6m 47s | the patch passed with JDK v1.8.0_72 |
| +1 | javac | 6m 47s | the patch passed |
| +1 | compile | 7m 34s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 7m 34s | the patch passed |
| -1 | checkstyle | 0m 26s | hadoop-common-project/hadoop-common: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139) |
| +1 | mvnsite | 1m 15s | the patch passed |
| +1 | mvneclipse | 0m 15s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 11s | the patch passed |
| +1 | javadoc | 1m 3s | the patch passed with JDK v1.8.0_72 |
| +1 | javadoc | 1m 10s | the patch passed with JDK v1.7.0_95 |
| +1 | unit | 9m 34s | hadoop-common in the patch passed with JDK v1.8.0_72. |
| +1 | unit | 9m 15s | hadoop-common in the patch passed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 26s | Patch does not generate ASF License warnings. |
| 68m 35s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12788139/c10865_20160216.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux c9a188397cee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4b0e59f |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
| JDK v1.7.0_95 Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8637/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks for the update, Tsz Wo Nicholas Sze. TestDataChecksum#testBulkOps fails in non-native profile. There seems to be a problem in calculation of errPos in DataChecksum#verifyChunked.
java.lang.AssertionError: expected:<512> but was:<511>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at org.apache.hadoop.util.TestDataChecksum$Harness.testCorrectness(TestDataChecksum.java:120)
at org.apache.hadoop.util.TestDataChecksum.doBulkTest(TestDataChecksum.java:132)
at org.apache.hadoop.util.TestDataChecksum.testBulkOps(TestDataChecksum.java:51)
Good catch! The errPos was wrong.
c10865_20160216b.patch: fixes the bug.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 10s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 7m 2s | trunk passed |
| +1 | compile | 7m 7s | trunk passed with JDK v1.8.0_72 |
| +1 | compile | 7m 21s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 23s | trunk passed |
| +1 | mvnsite | 1m 9s | trunk passed |
| +1 | mvneclipse | 0m 14s | trunk passed |
| +1 | findbugs | 1m 42s | trunk passed |
| +1 | javadoc | 0m 58s | trunk passed with JDK v1.8.0_72 |
| +1 | javadoc | 1m 7s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 43s | the patch passed |
| +1 | compile | 6m 52s | the patch passed with JDK v1.8.0_72 |
| +1 | javac | 6m 52s | the patch passed |
| +1 | compile | 7m 31s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 7m 31s | the patch passed |
| -1 | checkstyle | 0m 24s | hadoop-common-project/hadoop-common: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139) |
| +1 | mvnsite | 1m 7s | the patch passed |
| +1 | mvneclipse | 0m 14s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 1m 53s | the patch passed |
| +1 | javadoc | 0m 56s | the patch passed with JDK v1.8.0_72 |
| +1 | javadoc | 1m 10s | the patch passed with JDK v1.7.0_95 |
| +1 | unit | 8m 24s | hadoop-common in the patch passed with JDK v1.8.0_72. |
| +1 | unit | 8m 21s | hadoop-common in the patch passed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 22s | Patch does not generate ASF License warnings. |
| 66m 20s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12788159/c10865_20160216b.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 73a5f2ee29e2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 77ba5ad |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
| JDK v1.7.0_95 Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/8638/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Hmm.. TestDFSClientRetries#testRetryOnChecksumFailure expects that the description of ChecksumException contains "Checksum error". (Jenkins did not run the test because it is under hadoop-hdfs.)
java.lang.AssertionError: Expected to find 'Checksum error' but got unexpected exception:org.apache.hadoop.fs.ChecksumException: Checksum CRC32C not matched for file /corrupted at position 0: expected=3047312A but computed=B5E7A6EF, algorithm=PureJavaCrc32C
...
at org.apache.hadoop.hdfs.TestDFSClientRetries.testRetryOnChecksumFailure(TestDFSClientRetries.java:930)
The assertion or DataChecksum#throwChecksumException should be fixed to match these.
try {
dis.read(arr, 0, (int)FILE_LENGTH);
fail("Expected ChecksumException not thrown");
} catch (Exception ex) {
GenericTestUtils.assertExceptionContains(
"Checksum error", ex);
}
}
Thanks. Here is a patch fixing TestDFSClientRetries.
c10865_20160217.patch
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 9s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
| 0 | mvndep | 0m 34s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 49s | trunk passed |
| +1 | compile | 6m 55s | trunk passed with JDK v1.8.0_72 |
| +1 | compile | 6m 50s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 1m 6s | trunk passed |
| +1 | mvnsite | 1m 56s | trunk passed |
| +1 | mvneclipse | 0m 28s | trunk passed |
| +1 | findbugs | 3m 29s | trunk passed |
| +1 | javadoc | 2m 4s | trunk passed with JDK v1.8.0_72 |
| +1 | javadoc | 3m 0s | trunk passed with JDK v1.7.0_95 |
| 0 | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 31s | the patch passed |
| +1 | compile | 6m 17s | the patch passed with JDK v1.8.0_72 |
| +1 | javac | 6m 17s | the patch passed |
| +1 | compile | 6m 46s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 6m 46s | the patch passed |
| -1 | checkstyle | 1m 5s | root: patch generated 1 new + 132 unchanged - 7 fixed = 133 total (was 139) |
| +1 | mvnsite | 1m 55s | the patch passed |
| +1 | mvneclipse | 0m 27s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 3m 57s | the patch passed |
| +1 | javadoc | 2m 12s | the patch passed with JDK v1.8.0_72 |
| +1 | javadoc | 2m 51s | the patch passed with JDK v1.7.0_95 |
| +1 | unit | 7m 13s | hadoop-common in the patch passed with JDK v1.8.0_72. |
| -1 | unit | 56m 34s | hadoop-hdfs in the patch failed with JDK v1.8.0_72. |
| +1 | unit | 7m 58s | hadoop-common in the patch passed with JDK v1.7.0_95. |
| -1 | unit | 53m 41s | hadoop-hdfs in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 30s | Patch does not generate ASF License warnings. |
| 188m 1s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_72 Failed junit tests | hadoop.hdfs.TestFileAppend |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
This message was automatically generated.
+1. The failure of TestFileAppend and TestShortCircuitCache is not related to the patch.
Thanks Masatake Iwasaki for reviewing and testing the patches!
I have committed this.
FAILURE: Integrated in Hadoop-trunk-Commit #9324 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9324/)
HADOOP-10865. Add a Crc32 chunked verification benchmark for both (szetszwo: rev bbfaf3c2712c9ba82b0f8423bdeb314bf505a692)
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
- hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDataChecksum.java
- hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/Crc32PerformanceTest.java
c10865_20140717.patch: copied from HADOOP-10778.