Details
-
Type:
Sub-task
-
Status: Patch Available
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: HDFS-10467
-
Component/s: fs
-
Labels:None
Description
Add the Web UI to the Router to expose the status of the federated cluster. It includes the federation metrics.
Issue Links
- depends upon
-
HDFS-12335
Federation Metrics
-
- Resolved
-
- relates to
-
HDFS-12510
RBF: Add security to UI
-
- Open
-
-
HDFS-12512
RBF: Add WebHDFS
-
- Open
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Really minor:
- If the comparator isn't used except for JSON output, it can move out of MountTable
+0, overall. I can offer no valuable input on web UIs. Ravi Prakash, do you want to take a look?
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 20s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
| HDFS-10467 Compile Tests | |||
| +1 | mvninstall | 19m 38s | HDFS-10467 passed |
| +1 | compile | 0m 55s | HDFS-10467 passed |
| +1 | checkstyle | 0m 46s | HDFS-10467 passed |
| +1 | mvnsite | 1m 4s | HDFS-10467 passed |
| -1 | findbugs | 2m 1s | hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings. |
| +1 | javadoc | 0m 49s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 56s | the patch passed |
| +1 | compile | 0m 56s | the patch passed |
| -1 | javac | 0m 56s | hadoop-hdfs-project_hadoop-hdfs generated 11 new + 400 unchanged - 9 fixed = 411 total (was 409) |
| -0 | checkstyle | 0m 44s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 403 unchanged - 0 fixed = 416 total (was 403) |
| +1 | mvnsite | 1m 2s | the patch passed |
| -1 | whitespace | 0m 0s | The patch 36 line(s) with tabs. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| -1 | findbugs | 2m 9s | hadoop-hdfs-project/hadoop-hdfs generated 3 new + 10 unchanged - 0 fixed = 13 total (was 10) |
| +1 | javadoc | 0m 50s | the patch passed |
| Other Tests | |||
| -1 | unit | 75m 22s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 109m 26s |
| Reason | Tests |
|---|---|
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getNamespaceInfo(String) At FederationMetrics.java:is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getNamespaceInfo(String) At FederationMetrics.java:[line 509] | |
| Call to method of static java.text.DateFormat in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getDateString(long) At FederationMetrics.java:java.text.DateFormat in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getDateString(long) At FederationMetrics.java:[line 644] | |
| Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getNamespaceInfo(String) At NamenodeBeanMetrics.java:is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getNamespaceInfo(String) At NamenodeBeanMetrics.java:[line 387] | |
| Failed junit tests | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.tools.TestHdfsConfigFields |
This message was automatically generated.
Fixed most Jenkins comments.
I'm leaving the comparator because it will be used by the Rebalancer output.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 13s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
| HDFS-10467 Compile Tests | |||
| 0 | mvndep | 1m 43s | Maven dependency ordering for branch |
| +1 | mvninstall | 17m 15s | HDFS-10467 passed |
| +1 | compile | 17m 52s | HDFS-10467 passed |
| +1 | checkstyle | 2m 20s | HDFS-10467 passed |
| +1 | mvnsite | 3m 11s | HDFS-10467 passed |
| -1 | findbugs | 2m 18s | hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings. |
| +1 | javadoc | 1m 59s | HDFS-10467 passed |
| Patch Compile Tests | |||
| 0 | mvndep | 0m 18s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 57s | the patch passed |
| +1 | compile | 13m 2s | the patch passed |
| -1 | javac | 13m 2s | root generated 9 new + 1329 unchanged - 9 fixed = 1338 total (was 1338) |
| -0 | checkstyle | 2m 21s | root: The patch generated 4 new + 403 unchanged - 0 fixed = 407 total (was 403) |
| +1 | mvnsite | 2m 56s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 5s | The patch has no ill-formed XML file. |
| -1 | findbugs | 2m 36s | hadoop-hdfs-project/hadoop-hdfs generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) |
| +1 | javadoc | 2m 1s | the patch passed |
| Other Tests | |||
| +1 | unit | 8m 38s | hadoop-common in the patch passed. |
| -1 | unit | 67m 50s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| 154m 17s |
| Reason | Tests |
|---|---|
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:[lines 167-168] | |
| Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:[lines 197-199] | |
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| hadoop.hdfs.TestFileCorruption | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
This message was automatically generated.
I can try to take a look but honestly I'm sorry it won't have very high priority.
I didn't realize this was in the Federation branch. This is way out of my field. Sorry. Could some branch committers please review it?
Ravi Prakash, we would appreciate feedback on the pure UI side that tries to mimic the Namenode UI.
Íñigo Goiri it would be great if you add some screenshots of the UI.
Screenshots from development cluster.
Removed some potentially confidential information.
Rebased to HDFS-10467 and fixed a couple minor issues.
There are a bunch of warnings/checkstyles/findbugs that are the same for NameNode.
Open to ideas on how to fix those.
Thanks Íñigo Goiri. Can we split this patch in 2 Jiras - UI and metrics?
I have already created HDFS-12335 for metrics.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 17s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
| HDFS-10467 Compile Tests | |||
| -1 | mvninstall | 15m 47s | root in HDFS-10467 failed. |
| +1 | compile | 0m 53s | HDFS-10467 passed |
| +1 | checkstyle | 0m 42s | HDFS-10467 passed |
| +1 | mvnsite | 0m 59s | HDFS-10467 passed |
| +1 | findbugs | 1m 52s | HDFS-10467 passed |
| +1 | javadoc | 0m 48s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 54s | the patch passed |
| +1 | compile | 0m 51s | the patch passed |
| -1 | javac | 0m 51s | hadoop-hdfs-project_hadoop-hdfs generated 10 new + 413 unchanged - 10 fixed = 423 total (was 423) |
| -0 | checkstyle | 0m 39s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 403 unchanged - 0 fixed = 406 total (was 403) |
| +1 | mvnsite | 0m 58s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 3s | The patch has no ill-formed XML file. |
| -1 | findbugs | 1m 59s | hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 | javadoc | 0m 53s | the patch passed |
| Other Tests | |||
| -1 | unit | 84m 31s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| 114m 6s |
| Reason | Tests |
|---|---|
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:[lines 167-168] | |
| Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:[lines 197-199] | |
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| hadoop.hdfs.TestDFSShell |
This message was automatically generated.
Giovanni Matteo Fumarola, thank you for the comments.
I pushed the metrics to HDFS-12335.
Rebased after HDFS-12335.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 17s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
| HDFS-10467 Compile Tests | |||
| +1 | mvninstall | 15m 1s | HDFS-10467 passed |
| +1 | compile | 0m 52s | HDFS-10467 passed |
| +1 | checkstyle | 0m 44s | HDFS-10467 passed |
| +1 | mvnsite | 0m 57s | HDFS-10467 passed |
| +1 | findbugs | 1m 55s | HDFS-10467 passed |
| +1 | javadoc | 0m 48s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 51s | the patch passed |
| +1 | compile | 0m 50s | the patch passed |
| -1 | javac | 0m 50s | hadoop-hdfs-project_hadoop-hdfs generated 5 new + 403 unchanged - 5 fixed = 408 total (was 408) |
| +1 | checkstyle | 0m 40s | the patch passed |
| +1 | mvnsite | 0m 53s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 | findbugs | 1m 53s | the patch passed |
| +1 | javadoc | 0m 44s | the patch passed |
| Other Tests | |||
| -1 | unit | 94m 12s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 122m 21s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestLeaseRecoveryStriped |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 | |
| hadoop.hdfs.TestParallelShortCircuitReadUnCached | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.tools.TestHdfsConfigFields | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 | |
| hadoop.hdfs.TestListFilesInFileContext | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12273 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12886116/HDFS-12273-HDFS-10467-003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle |
| uname | Linux 5437e609bbe6 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10467 / d522007 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | https://builds.apache.org/job/PreCommit-HDFS-Build/21054/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21054/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21054/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The javac errors are not related to this commit; I've rebased HDFS-10467 to get rid of some of the unit tests.
Giovanni Matteo Fumarola, do you mind taking a look?
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
| HDFS-10467 Compile Tests | |||
| +1 | mvninstall | 17m 3s | HDFS-10467 passed |
| +1 | compile | 1m 1s | HDFS-10467 passed |
| +1 | checkstyle | 0m 46s | HDFS-10467 passed |
| +1 | mvnsite | 1m 0s | HDFS-10467 passed |
| +1 | findbugs | 2m 4s | HDFS-10467 passed |
| +1 | javadoc | 0m 47s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 1m 4s | the patch passed |
| +1 | compile | 0m 57s | the patch passed |
| -1 | javac | 0m 57s | hadoop-hdfs-project_hadoop-hdfs generated 5 new + 403 unchanged - 5 fixed = 408 total (was 408) |
| +1 | checkstyle | 0m 42s | the patch passed |
| +1 | mvnsite | 0m 58s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 | findbugs | 2m 24s | the patch passed |
| +1 | javadoc | 0m 57s | the patch passed |
| Other Tests | |||
| -1 | unit | 91m 20s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| 123m 42s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| hadoop.hdfs.TestLeaseRecoveryStriped | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 | |
| hadoop.hdfs.server.federation.router.TestNamenodeHeartbeat | |
| hadoop.hdfs.TestReconstructStripedFile | |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12273 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12886991/HDFS-12273-HDFS-10467-004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle |
| uname | Linux ea1d8202a746 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10467 / 679e31a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac | https://builds.apache.org/job/PreCommit-HDFS-Build/21128/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21128/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21128/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21128/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The failure in TestNamenodeHeartbeat is not related to this change.
I opened HDFS-12450 to fix it.
Skimming the smaller patch, most of it looks straightforward.
I'm not familiar with how the new UIs protect against XSS vulnerabilities. The HtmlQuoting class used to take care of this for the old UI, IIRC Hamlet has its own strategy... Haohui Mai?
XSS handling could also be part of HDFS-12284. If we defer that, then it might make sense to add a note to the docs, so users know that security is not supported and not audited.
Thanks Chris Douglas for the check.
I'm OK with handling security issues related to the Web UI in HDFS-12284 or maybe a follow-up JIRA.
xiangguang zheng, Zhe Zhang, does it make sense to do it in HDFS-12284?
Regarding the note in the doc, I could add it here or HDFS-12381 as a general comment on security and not only about the Web UI.
Sorry I've since swapped out all front-end and security knowledge from my brain. But for the new UI, I had to add https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java#L315 . My guide were the things done for onOpen / onAppend etc. XSS isn't quite related to HDFS-12284, so if at all you want to postpone the analysis, would it make sense to file a different JIRA?
Regarding the note in the doc, I could add it here or HDFS-12381 as a general comment on security and not only about the Web UI.
Adding it to HDFS-12381 is simplest.
XSS isn't quite related to HDFS-12284, so if at all you want to postpone the analysis, would it make sense to file a different JIRA?
Sure, fair enough. Even if "harden the federation UI" is closed without requiring any code, it'd be useful for tracking.
If we defer hardening the UI to after the merge, the current patch seems fine to me.
Thanks Ravi Prakash, actually, one thing is missing is the WebHDFS interface.
Do you guys think it should go in HDFS-10467?
In any case, that would require the web app from this JIRA.
one thing is missing is the WebHDFS interface. Do you guys think it should go in HDFS-10467?
That could go in after the merge pretty easily. Certainly not a prerequisite.
Ravi Prakash, if it's not too traumatic for you to review/relive the front-end and security bits, then we can wait on your review. Otherwise we make a concerted effort to harden this code later, after HDFS-12284 is in.
I eyeballed the much smaller patch and it seems fine to me. Let's please file the JIRA any way, and we can pore over this and the trunk code altogether just once.
Ravi Prakash, sounds good.
I'll create those JIRAs outside of the HDFS-10467 umbrella for the future merge.
Thinking on following the naming RBF.
I created HDFS-12510 for adding security to the web server and HDFS-12512 to add support for WebHDFS.
Web UI with metrics.