Details
-
Type:
Sub-task
-
Status: Patch Available
-
Priority:
Minor
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: HDFS-10467
-
Component/s: fs
-
Labels:None
Description
Adding configuration options in tabular format.
Issue Links
- depends upon
-
HDFS-11554
[Documentation] Router-based federation documentation
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 28s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| HDFS-10467 Compile Tests | |||
| -1 | mvninstall | 15m 53s | root in HDFS-10467 failed. |
| +1 | mvnsite | 1m 8s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvnsite | 0m 59s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| Other Tests | |||
| +1 | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| 19m 2s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12381 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884855/HDFS-12381-HDFS-10467.000.patch |
| Optional Tests | asflicense mvnsite |
| uname | Linux d770de1e4e4a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10467 / fc2c254 |
| mvninstall | https://builds.apache.org/job/PreCommit-HDFS-Build/20959/artifact/patchprocess/branch-mvninstall-root.txt |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20959/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Given the comment from Manoj Govindassamy about recommendations/best-practices about setting up the mount table, I think I can extend this JIRA to cover this a little more.
Thanks Inigo. Previously attached patch has good details. Additional config related items will be very useful.
Íñigo Goiri thanks for reporting and working on this.
Minor nits:
1) Please correct the typo's
182 | Propery | Description|
2) How about chage like "If true ,the Router should monitor the namenode in the local machine" which will be sync with existing conf .If you agree, please change hdfs-default.xml and remaining configs also.
| dfs.federation.router.monitor.localnamenode.enable | If the Router should monitor the namenode in the local machine. |
3) Are you going update with this some more details..?
recommendations/best-practices about setting up the mount table, I think I can extend this JIRA to cover this a little more.
Thanks Brahma Reddy Battula for the comments.
I pushed a new patch with most of them.
Not super proud of the best-practices part so I'm open to suggestions.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 17s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| HDFS-10467 Compile Tests | |||
| -1 | mvninstall | 4m 22s | root in HDFS-10467 failed. |
| -1 | compile | 0m 29s | hadoop-hdfs in HDFS-10467 failed. |
| -1 | mvnsite | 0m 29s | hadoop-hdfs in HDFS-10467 failed. |
| +1 | javadoc | 0m 46s | HDFS-10467 passed |
| Patch Compile Tests | |||
| -1 | mvninstall | 0m 28s | hadoop-hdfs in the patch failed. |
| -1 | compile | 0m 28s | hadoop-hdfs in the patch failed. |
| -1 | javac | 0m 28s | hadoop-hdfs in the patch failed. |
| -1 | mvnsite | 0m 29s | hadoop-hdfs in the patch failed. |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 47s | the patch passed |
| Other Tests | |||
| -1 | unit | 0m 30s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| 10m 35s |
This message was automatically generated.
The errors are caused by modifying hdfs-default.xml.
I'll ignore the output from jenkins.
There are compilations errors , can you look once? I am on mobile.Errors might not related to this jira.
The error is:
RouterRpcServer.java:[138,7] error: RouterRpcServer is not abstract and does not override abstract method getECBlockGroupStats() in ClientProtocol
Apparently there is yet one more method addition to ClientProtocol related to EC.
This is weird because HDFS-10467 was working and I haven't rebased.
I'll post another rebase JIRA for HDFS-10467.
Apparently there is yet one more method addition to ClientProtocol related to EC.
Yes, it's after HDFS-12381. we need to rebase.
Compilation Errors for reference
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project h adoop-hdfs: Compilation failure: Compilation failure: [ERROR] /D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou ter/RouterRpcServer.java:[69,39] cannot find symbol [ERROR] symbol: class BlocksStats [ERROR] location: package org.apache.hadoop.hdfs.protocol [ERROR] /D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou ter/RouterRpcServer.java:[79,39] cannot find symbol [ERROR] symbol: class ECBlockGroupsStats [ERROR] location: package org.apache.hadoop.hdfs.protocol [ERROR] /D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou ter/RouterRpcServer.java:[1882,10] cannot find symbol [ERROR] symbol: class ECBlockGroupsStats [ERROR] location: class org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer [ERROR] /D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou ter/RouterRpcServer.java:[1894,10] cannot find symbol [ERROR] symbol: class BlocksStats [ERROR] location: class org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer [ERROR] /D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou ter/RouterRpcServer.java:[138,8] org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer is not abstract and doe s not override abstract method getECBlockGroupStats() in org.apache.hadoop.hdfs.protocol.ClientProtocol [ERROR] /D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou ter/RouterRpcServer.java:[1881,3] method does not override or implement a method from a supertype [ERROR] /D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou ter/RouterRpcServer.java:[1893,3] method does not override or implement a method from a supertype
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 23s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| HDFS-10467 Compile Tests | |||
| +1 | mvninstall | 21m 4s | HDFS-10467 passed |
| +1 | compile | 1m 10s | HDFS-10467 passed |
| +1 | mvnsite | 1m 18s | HDFS-10467 passed |
| +1 | javadoc | 1m 0s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 1m 13s | the patch passed |
| +1 | compile | 0m 55s | the patch passed |
| +1 | javac | 0m 55s | the patch passed |
| +1 | mvnsite | 0m 54s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 43s | the patch passed |
| Other Tests | |||
| -1 | unit | 98m 58s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 24s | The patch does not generate ASF License warnings. |
| 129m 19s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFile |
| hadoop.hdfs.TestLeaseRecoveryStriped | |
| hadoop.hdfs.TestWriteRead | |
| hadoop.hdfs.TestSafeModeWithStripedFile | |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.TestSeekBug | |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicy | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration | |
| hadoop.hdfs.server.namenode.ha.TestHASafeMode | |
| hadoop.hdfs.server.namenode.TestReencryptionWithKMS | |
| hadoop.hdfs.server.federation.router.TestNamenodeHeartbeat | |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12381 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12886990/HDFS-12381-HDFS-10467.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
| uname | Linux 71f39e4f7663 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10467 / 679e31a |
| Default Java | 1.8.0_144 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21130/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21130/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1 lgtm. Brahma Reddy Battula, Manoj Govindassamy: any other feedback?
Íñigo Goiri thanks for updating the patch.
I am thinking following also,sorry to trouble you.
1) can we add stop command for router..? may be start command also update like " hdfs --daemon start router"..?
2) how about adding default value to each property in the table(so totally three colums)..? and change description remaining boolean as i mentioned in earlier comment..?
3) Fix following two typos also,like below..?
Advanced functions like snapshotting, encryption
Advanced functions like snapshot, encryption
Adminstrators can query information
Administrators can query information
#2 and #3 sound good.
For #1, you are talking about adding this in the documentation?
If so, that's OK with me.
In the deployment section or somewhere else?
Feel free to ask for any other documentation addition.
I've been heads down with this for a while and everything seems obvious to me and I'm assuming a lot of things.
The easier it is to understand, the more chances for this to be adopted.
Overall LGTM. +1.
For example, if we want a federated address called `/data/wl1`, it is recommended to have that same name in the destination namespace.
- Federated address is not consistent with the wordings 'federated folder' mentioned in the previous line. Also for destination namespace. Either we can call it address by including the schema, authority, etc., or may be call it just federated address? Your thoughts?
- It would be really helpful if we can touch upon the non-existing mount points as well.
Thanks for the comments Brahma Reddy Battula and Manoj Govindassamy.
From your comments, I realized that I wasn't talking much about the client, I added a full section on that.
At the same time, I think I covered most of your comments.
Let me know if there is anything else you think should be documented.
Chris Douglas I added a note on security being work in progress.
Does that seems reasonable as a placeholder until HDFS-12284 is complete?
I added a note on security being work in progress.
For user docs, might as well spell it out instead of linking into JIRA. I'd also move the notice closer to the top of the page, rather than the bottom. Something like:
Secure authentication and authorization are not supported yet, so the Router will not proxy to Hadoop clusters with security enabled.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 2m 56s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| HDFS-10467 Compile Tests | |||
| +1 | mvninstall | 14m 13s | HDFS-10467 passed |
| +1 | compile | 0m 52s | HDFS-10467 passed |
| +1 | mvnsite | 0m 56s | HDFS-10467 passed |
| +1 | javadoc | 0m 53s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 57s | the patch passed |
| +1 | compile | 0m 48s | the patch passed |
| +1 | javac | 0m 48s | the patch passed |
| +1 | mvnsite | 0m 55s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 43s | the patch passed |
| Other Tests | |||
| -1 | unit | 96m 10s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 17s | The patch does not generate ASF License warnings. |
| 120m 54s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| hadoop.hdfs.TestLeaseRecoveryStriped | |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12381 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12887455/HDFS-12381-HDFS-10467.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
| uname | Linux a02b1f7279a6 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10467 / 2d490d3 |
| Default Java | 1.8.0_144 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21179/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21179/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21179/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1 lgtm
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| HDFS-10467 Compile Tests | |||
| +1 | mvninstall | 18m 34s | HDFS-10467 passed |
| +1 | compile | 0m 50s | HDFS-10467 passed |
| +1 | mvnsite | 0m 57s | HDFS-10467 passed |
| +1 | javadoc | 0m 46s | HDFS-10467 passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 51s | the patch passed |
| +1 | compile | 0m 48s | the patch passed |
| +1 | javac | 0m 48s | the patch passed |
| +1 | mvnsite | 0m 52s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 44s | the patch passed |
| Other Tests | |||
| -1 | unit | 83m 21s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
| 109m 31s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestHFlush |
| hadoop.hdfs.TestLeaseRecoveryStriped | |
| hadoop.hdfs.server.namenode.TestCacheDirectives | |
| hadoop.hdfs.TestClientProtocolForPipelineRecovery | |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12381 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12887532/HDFS-12381-HDFS-10467.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
| uname | Linux c54443bb85a4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10467 / 2d490d3 |
| Default Java | 1.8.0_144 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21186/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21186/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21186/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Test Failures are unrelated,me too +1 on latest patch.
Manoj Govindassamy, do you have any other comments?
As Brahma Reddy Battula suggested, I will add a tabular with all the configuration options.