Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha2
-
Labels:None
-
Hadoop Flags:Reviewed
Description
Webhdfs is extremely expensive under load and is not subject to the QoS benefits of the RPC call queue. HADOOP-13537 provides the basis for routing webhdfs through the call queue to provide unified QoS.
Issue Links
- relates to
-
HDFS-11000
webhdfs PUT does not work if requests are routed to call queue.
-
- Resolved
-
- requires
-
HADOOP-13537
Support external calls in the RPC call queue
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
The current situation is all the jetty handler threads end up competing unfairly with the ipc handlers. The unregulated competition easily causes callq overflow and rpc calls are rejected. The jetty acceptor is still busy accepting an unlimited number of connections even while all the jetty handlers are blocked. A full GC is often inevitable when hit with an abusive surge.
With this patch, the callq can provide fairness. Also assuming there's at least one jetty handler not blocked in the namesystem, new requests will be rejected and their connections will be closed.
The patch does not apply anymore. Please rebase/update.
Attaching rebased patch.
Thanks much for working on this Daryn Sharp Kihwal Lee and Rushabh S Shah.
One feature we want is for WebHDFS calls to be restricted under the RPC ACLs (e.g. revoking a certain user's permission to issue certain WebHDFS calls). I haven't look into much detail of the patch. Any quick answer on this? Thanks!
The ACL authorization is currently only checked when processing the IPC connection context as part of the connection lifecycle prior to reading calls. Webhdfs has no connection context. I think applying common ACLs to webhdfs would be a useful feature but as a separate jira. Most likely it should be a change in the webhdfs servlet.
The RPC ACL controls at protocol level, bit it looks like what you want is method level restrictions. It will best to open a new jira and present use cases to facilitate further discussions.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 17s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 6m 58s | trunk passed |
| +1 | compile | 0m 46s | trunk passed |
| +1 | checkstyle | 0m 37s | trunk passed |
| +1 | mvnsite | 0m 50s | trunk passed |
| +1 | mvneclipse | 0m 12s | trunk passed |
| +1 | findbugs | 1m 42s | trunk passed |
| +1 | javadoc | 0m 55s | trunk passed |
| +1 | mvninstall | 0m 48s | the patch passed |
| +1 | compile | 0m 43s | the patch passed |
| +1 | javac | 0m 43s | the patch passed |
| -0 | checkstyle | 0m 34s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 917 unchanged - 5 fixed = 925 total (was 922) |
| +1 | mvnsite | 0m 49s | the patch passed |
| +1 | mvneclipse | 0m 10s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 49s | the patch passed |
| +1 | javadoc | 0m 52s | the patch passed |
| -1 | unit | 60m 13s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
| 79m 52s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.TestDFSShell | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.web.TestWebHDFSXAttr |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12831389/HDFS-10789-1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux f561fce8f501 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ef7f06f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16997/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16997/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16997/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16997/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
| 0 | mvndep | 0m 18s | Maven dependency ordering for branch |
| +1 | mvninstall | 7m 7s | trunk passed |
| +1 | compile | 6m 46s | trunk passed |
| +1 | checkstyle | 1m 38s | trunk passed |
| +1 | mvnsite | 1m 47s | trunk passed |
| +1 | mvneclipse | 0m 25s | trunk passed |
| +1 | findbugs | 2m 56s | trunk passed |
| +1 | javadoc | 1m 20s | trunk passed |
| 0 | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 19s | the patch passed |
| +1 | compile | 6m 42s | the patch passed |
| +1 | javac | 6m 42s | the patch passed |
| -0 | checkstyle | 1m 38s | root: The patch generated 9 new + 971 unchanged - 5 fixed = 980 total (was 976) |
| +1 | mvnsite | 1m 44s | the patch passed |
| +1 | mvneclipse | 0m 25s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | findbugs | 3m 13s | the patch passed |
| +1 | javadoc | 1m 21s | the patch passed |
| +1 | unit | 7m 14s | hadoop-common in the patch passed. |
| -1 | unit | 57m 6s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
| 104m 46s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832730/HDFS-10789-2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
| uname | Linux 5276a9b6dac2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2fb392a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/17106/artifact/patchprocess/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17106/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17106/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17106/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Cancelling to address checkstyle issues.
TestNameNodeMetadataConsistency#testGenerationStampInFuture test fails intermittently and is tracked by HDFS-10499.
forgot to address checkstyle in NameNodeRpcServer.java file
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 13s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
| 0 | mvndep | 1m 33s | Maven dependency ordering for branch |
| +1 | mvninstall | 7m 27s | trunk passed |
| +1 | compile | 7m 44s | trunk passed |
| +1 | checkstyle | 1m 40s | trunk passed |
| +1 | mvnsite | 2m 0s | trunk passed |
| +1 | mvneclipse | 0m 28s | trunk passed |
| +1 | findbugs | 3m 11s | trunk passed |
| +1 | javadoc | 1m 25s | trunk passed |
| 0 | mvndep | 0m 17s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 23s | the patch passed |
| +1 | compile | 6m 42s | the patch passed |
| +1 | javac | 6m 42s | the patch passed |
| -0 | checkstyle | 1m 38s | root: The patch generated 2 new + 971 unchanged - 5 fixed = 973 total (was 976) |
| +1 | mvnsite | 1m 43s | the patch passed |
| +1 | mvneclipse | 0m 25s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 | findbugs | 3m 12s | the patch passed |
| +1 | javadoc | 1m 20s | the patch passed |
| -1 | unit | 21m 7s | hadoop-common in the patch failed. |
| +1 | unit | 58m 21s | hadoop-hdfs in the patch passed. |
| +1 | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| 123m 4s |
| Reason | Tests |
|---|---|
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832777/HDFS-10789-4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
| uname | Linux ee659129fbde 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8a09bf7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/17112/artifact/patchprocess/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17112/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17112/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17112/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
I missed 2 checkstyle warnings again. 
Attaching a patch fixing 2 checkstyle warnings.
Hopefully the last patch.
 +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 12s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
| 0 | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 32s | trunk passed |
| +1 | compile | 6m 46s | trunk passed |
| +1 | checkstyle | 1m 38s | trunk passed |
| +1 | mvnsite | 1m 48s | trunk passed |
| +1 | mvneclipse | 0m 26s | trunk passed |
| +1 | findbugs | 2m 58s | trunk passed |
| +1 | javadoc | 1m 20s | trunk passed |
| 0 | mvndep | 0m 13s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 20s | the patch passed |
| +1 | compile | 6m 43s | the patch passed |
| +1 | javac | 6m 43s | the patch passed |
| +1 | checkstyle | 1m 38s | root: The patch generated 0 new + 971 unchanged - 5 fixed = 971 total (was 976) |
| +1 | mvnsite | 1m 43s | the patch passed |
| +1 | mvneclipse | 0m 25s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | findbugs | 3m 14s | the patch passed |
| +1 | javadoc | 1m 20s | the patch passed |
| +1 | unit | 7m 59s | hadoop-common in the patch passed. |
| +1 | unit | 57m 41s | hadoop-hdfs in the patch passed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 105m 24s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832925/HDFS-10789-5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
| uname | Linux 83f74f2f68fe 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6476934 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17122/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17122/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1 looks good. We have been running this internally for quite some time (over 6 months).
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10597 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10597/)
HDFS-10789. Route webhdfs through the RPC call queue. Contributed by (kihwal: rev 85cd06f6636f295ad1f3bf2a90063f4714c9cca7)
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
- (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
- (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Committed this to trunk, branch-2 and branch-2.8.
Thanks for providing the original patch, Daryn. Also thanks Rushabh for driving this to the end.
Thanks Kihwal Lee for the reviews and commit.
I remember we had to fix one bug after this feature. Filed HDFS-11000.
What happens when the call queue is full and backoff is on? What does the http client get and what does WebHdfsFilesSystem do in terms of retry? Does the connection go away or can they still pile up when the webhdfs load is high?