Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
java.lang.RuntimeException: org.apache.hadoop.ipc.DecayRpcScheduler could not be constructed. at org.apache.hadoop.ipc.CallQueueManager.createScheduler(CallQueueManager.java:89) at org.apache.hadoop.ipc.CallQueueManager.swapQueue(CallQueueManager.java:260) at org.apache.hadoop.ipc.Server.refreshCallQueue(Server.java:650) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.refreshCallQueue(NameNodeRpcServer.java:1582) at org.apache.hadoop.ipc.protocolPB.RefreshCallQueueProtocolServerSideTranslatorPB.refreshCallQueue(RefreshCallQueueProtocolServerSideTranslatorPB.java:49) at org.apache.hadoop.ipc.proto.RefreshCallQueueProtocolProtos$RefreshCallQueueProtocolService$2.callBlockingMethod(RefreshCallQueueProtocolProtos.java:769) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455) Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source DecayRpcSchedulerMetrics2.ipc.65110 already exists! at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:144) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:117)
Attachments
Attachments
- HADOOP-14207.006.patch
- 12 kB
- Surendra Singh Lilhore
- HADOOP-14207.005.patch
- 12 kB
- Surendra Singh Lilhore
- HADOOP-14207.004.patch
- 12 kB
- Surendra Singh Lilhore
- HADOOP-14207.003.patch
- 14 kB
- Surendra Singh Lilhore
- HADOOP-14207.002.patch
- 14 kB
- Surendra Singh Lilhore
- HADOOP-14207.001.patch
- 4 kB
- Surendra Singh Lilhore
Issue Links
- is blocked by
-
HADOOP-12916 Allow RPC scheduler/callqueue backoff using response times
- Resolved
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 17s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | mvninstall | 12m 28s | trunk passed |
+1 | compile | 19m 47s | trunk passed |
+1 | checkstyle | 0m 36s | trunk passed |
+1 | mvnsite | 1m 2s | trunk passed |
+1 | mvneclipse | 0m 19s | trunk passed |
+1 | findbugs | 1m 24s | trunk passed |
+1 | javadoc | 0m 49s | trunk passed |
+1 | mvninstall | 0m 37s | the patch passed |
+1 | compile | 15m 47s | the patch passed |
+1 | javac | 15m 47s | the patch passed |
+1 | checkstyle | 0m 36s | the patch passed |
+1 | mvnsite | 1m 2s | the patch passed |
+1 | mvneclipse | 0m 19s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | findbugs | 1m 36s | the patch passed |
+1 | javadoc | 0m 48s | the patch passed |
-1 | unit | 8m 29s | hadoop-common in the patch failed. |
+1 | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
68m 18s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.security.TestKDiag |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:a9ad5d6 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12859750/HADOOP-14207.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux f9843513ab78 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 2841666 |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/11865/artifact/patchprocess/whitespace-eol.txt |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11865/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11865/testReport/ |
modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11865/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks surendrasingh for reporting the issue and propose the fix. The patch looks good to me overall. Just a few comments:
DecayRpcScheduler.java
Line 236: we can remove this line and consolidate the registration inside MetricsProxy constructor().
Line 704: we should also unregister the MBeans similarly.
Can you add a test case in TestRefreshCallQueue with DecayRpcScheduler?
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 13s | Maven dependency ordering for branch |
+1 | mvninstall | 12m 42s | trunk passed |
+1 | compile | 21m 23s | trunk passed |
+1 | checkstyle | 1m 57s | trunk passed |
+1 | mvnsite | 2m 8s | trunk passed |
+1 | mvneclipse | 0m 39s | trunk passed |
+1 | findbugs | 3m 26s | trunk passed |
+1 | javadoc | 1m 41s | trunk passed |
0 | mvndep | 0m 16s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 32s | the patch passed |
+1 | compile | 16m 31s | the patch passed |
+1 | javac | 16m 31s | the patch passed |
-0 | checkstyle | 1m 58s | root: The patch generated 3 new + 78 unchanged - 6 fixed = 81 total (was 84) |
+1 | mvnsite | 2m 9s | the patch passed |
+1 | mvneclipse | 0m 47s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 49s | the patch passed |
+1 | javadoc | 1m 48s | the patch passed |
-1 | unit | 9m 3s | hadoop-common in the patch failed. |
-1 | unit | 68m 31s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 40s | The patch does not generate ASF License warnings. |
175m 57s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.security.TestKDiag |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:a9ad5d6 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12859912/HADOOP-14207.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux ca48d4378d9d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f462e1f |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/11874/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11874/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11874/testReport/ |
modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11874/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 13s | Maven dependency ordering for branch |
+1 | mvninstall | 12m 43s | trunk passed |
+1 | compile | 21m 17s | trunk passed |
+1 | checkstyle | 2m 4s | trunk passed |
+1 | mvnsite | 2m 13s | trunk passed |
+1 | mvneclipse | 0m 38s | trunk passed |
+1 | findbugs | 3m 36s | trunk passed |
+1 | javadoc | 1m 43s | trunk passed |
0 | mvndep | 0m 19s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 33s | the patch passed |
+1 | compile | 16m 2s | the patch passed |
+1 | javac | 16m 2s | the patch passed |
+1 | checkstyle | 1m 57s | root: The patch generated 0 new + 75 unchanged - 9 fixed = 75 total (was 84) |
+1 | mvnsite | 2m 3s | the patch passed |
+1 | mvneclipse | 0m 47s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 43s | the patch passed |
+1 | javadoc | 1m 46s | the patch passed |
-1 | unit | 8m 9s | hadoop-common in the patch failed. |
-1 | unit | 65m 7s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 42s | The patch does not generate ASF License warnings. |
171m 25s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.security.TestKDiag |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:a9ad5d6 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12860006/HADOOP-14207.003.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux a94eae1023b6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f462e1f |
Default Java | 1.8.0_121 |
findbugs | v3.0.0 |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11882/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11882/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11882/testReport/ |
modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11882/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Patch V3 looks good to me overall. Just have two more questions:
DecayRpcScheduler.java
Line 940: we should also remove the stale MetricsProxy instance from the static map (MetricsProxy.INSTANCES) so that the new MetricsProxy instance will be created for the newly configured call queue.
TestRefreshCallQueue.java
Can we reusing the existing per test setUp/tearDown to avoid duplicating for the two methods?
Thanks xyao for review..
Fixed both the comments. Attached v4 patch.
Please review...
Thanks for the update surendrasingh. We still need synchronized access of the INSTANCES map when removing. This can be wrapped as a static method of MetricsProxy with synchronized keyword.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 17s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 13s | Maven dependency ordering for branch |
+1 | mvninstall | 13m 19s | trunk passed |
+1 | compile | 16m 28s | trunk passed |
+1 | checkstyle | 1m 54s | trunk passed |
+1 | mvnsite | 2m 4s | trunk passed |
+1 | mvneclipse | 0m 42s | trunk passed |
-1 | findbugs | 1m 24s | hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. |
-1 | findbugs | 1m 46s | hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. |
+1 | javadoc | 1m 39s | trunk passed |
0 | mvndep | 0m 16s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 30s | the patch passed |
+1 | compile | 13m 37s | the patch passed |
+1 | javac | 13m 37s | the patch passed |
-0 | checkstyle | 2m 24s | root: The patch generated 1 new + 76 unchanged - 8 fixed = 77 total (was 84) |
+1 | mvnsite | 2m 11s | the patch passed |
+1 | mvneclipse | 0m 49s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 36s | the patch passed |
+1 | javadoc | 1m 48s | the patch passed |
-1 | unit | 7m 54s | hadoop-common in the patch failed. |
+1 | unit | 64m 13s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 44s | The patch does not generate ASF License warnings. |
163m 33s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.security.TestKDiag |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 22s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 44s | Maven dependency ordering for branch |
+1 | mvninstall | 15m 3s | trunk passed |
+1 | compile | 17m 35s | trunk passed |
+1 | checkstyle | 2m 1s | trunk passed |
+1 | mvnsite | 2m 17s | trunk passed |
+1 | mvneclipse | 0m 44s | trunk passed |
-1 | findbugs | 1m 39s | hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. |
-1 | findbugs | 1m 59s | hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. |
+1 | javadoc | 1m 41s | trunk passed |
0 | mvndep | 0m 16s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 34s | the patch passed |
+1 | compile | 15m 2s | the patch passed |
+1 | javac | 15m 2s | the patch passed |
+1 | checkstyle | 1m 57s | root: The patch generated 0 new + 76 unchanged - 8 fixed = 76 total (was 84) |
+1 | mvnsite | 2m 6s | the patch passed |
+1 | mvneclipse | 0m 49s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
-1 | findbugs | 1m 38s | hadoop-common-project/hadoop-common generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) |
+1 | javadoc | 1m 45s | the patch passed |
-1 | unit | 8m 0s | hadoop-common in the patch failed. |
+1 | unit | 64m 3s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
168m 31s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-common-project/hadoop-common |
DecayRpcScheduler$MetricsProxy is incompatible with expected argument type String in org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.removeInstance(DecayRpcScheduler$MetricsProxy) At DecayRpcScheduler.java:argument type String in org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.removeInstance(DecayRpcScheduler$MetricsProxy) At DecayRpcScheduler.java:[line 698] | |
Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
hadoop.security.TestKDiag |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
0 | mvndep | 0m 13s | Maven dependency ordering for branch |
+1 | mvninstall | 13m 27s | trunk passed |
+1 | compile | 15m 30s | trunk passed |
+1 | checkstyle | 1m 55s | trunk passed |
+1 | mvnsite | 2m 3s | trunk passed |
+1 | mvneclipse | 0m 42s | trunk passed |
-1 | findbugs | 1m 25s | hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. |
-1 | findbugs | 1m 45s | hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. |
+1 | javadoc | 1m 40s | trunk passed |
0 | mvndep | 0m 16s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 31s | the patch passed |
+1 | compile | 14m 9s | the patch passed |
+1 | javac | 14m 9s | the patch passed |
+1 | checkstyle | 1m 56s | root: The patch generated 0 new + 76 unchanged - 8 fixed = 76 total (was 84) |
+1 | mvnsite | 2m 8s | the patch passed |
+1 | mvneclipse | 0m 49s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 35s | the patch passed |
+1 | javadoc | 1m 47s | the patch passed |
-1 | unit | 7m 57s | hadoop-common in the patch failed. |
-1 | unit | 64m 22s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 42s | The patch does not generate ASF License warnings. |
162m 52s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
hadoop.security.TestKDiag | |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
hadoop.hdfs.TestMaintenanceState |
This message was automatically generated.
Thanks surendrasingh for the contribution. I commit the patch to trunk and branch-2.9
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11630 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11630/)
HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. (xyao: rev 4ea2778acb78f31b470e318d2b65ea43db76909b)
- (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java
- (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcScheduler.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
- (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
- (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DefaultRpcScheduler.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/TestRefreshCallQueue.java
xyao & surendrasingh,
I thought the target version for this is 2.8.1 (as mentioned in the jira's target version and as well as in wiki). Issue is applicable to 2.8.1 too right ?
Thansk Naganarasimha.
Issue is applicable to 2.8.1 too right ?
Yes, you're right. xyao can you commit this in 2.8.1 branch also ?
Thanks surendrasingh/Naganarasimha for the heads up. I have it cherry-picked in my local 2.8.1 branch last night. However, the build is failing with missing jars from apache repo. I was hesitated to push it up before figuring out the cause of it.
[ERROR] Failed to execute goal on project hadoop-common: Could not resolve dependencies for project org.apache.hadoop:hadoop-common:jar:2.8.1: Could not find artifact org.apache.hadoop:hadoop-auth:jar:tests:2.8.1 in apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots) -> [Help 1] [ERROR]
Thanks surendrasingh for working out a patch and xyao for review.
Reopen for 2.8.1. surendrasingh, can you provide a patch for 2.8.1 as well (patch name with branch-2.8.1 as postfix)?
junping_du and surendrasingh, the issue turns out to be an issue with my local repo. I will commit the cherry-pick to branch-2.8 and branch-2.8.1 soon.
2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
Attached initial patch ...
Please review...