Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.7.0
-
Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
-
Component/s: None
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
Opening this ticket on behalf of Daryn Sharp
HDFS-7435 introduced a more efficiently encoded block report format designed to improve performance and reduce GC load on the NN and DNs. The NN is not advertising this capability to the DNs so old-style reports are still being used.
Issue Links
- relates to
-
HDFS-9221
HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Jing Zhao: Daryn has a patch and he will upload on Monday.
I have to delete the previous comment since I copied the wrong user.
Any update here? I can also upload a patch to fix if Daryn Sharp is busy.
Yes, the enum is lazy initialized so it wasn't advertised. Here's the patch we've been using internally to force a static init.
Thanks for the fix, Daryn. +1 pending Jenkins.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 17m 38s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 55s | There were no new javac warning messages. |
| +1 | javadoc | 9m 49s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 1m 22s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 19s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 31s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 4s | Pre-build of native portion |
| -1 | hdfs tests | 171m 54s | Tests failed in hadoop-hdfs. |
| 216m 31s |
| Reason | Tests |
|---|---|
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12751090/HDFS-8867.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 71aedfa |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/12026/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/12026/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/12026/console |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 0m 8s | trunk compilation may be broken. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| -1 | javac | 0m 8s | The patch appears to cause the build to fail. |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12751090/HDFS-8867.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7ecbfd4 |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/12037/console |
This message was automatically generated.
Thanks for the fix, Daryn. I've committed this to trunk, branch-2 and branch-2.7.
FAILURE: Integrated in Hadoop-trunk-Commit #8322 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8322/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
FAILURE: Integrated in Hadoop-trunk-Commit #8323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8323/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #293 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/293/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
FAILURE: Integrated in Hadoop-Yarn-trunk #1023 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1023/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
ABORTED: Integrated in Hadoop-Hdfs-trunk #2220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2220/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #282 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/282/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/290/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2239 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2239/)
HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. (jing9: rev f61120d964a609ae5eabeb5c4d6c9afe0a15cad8)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java
Looks like the issue here is that CAPABILITIES_SUPPORTED has not been correctly initialized because the enum Capability is not loaded?