Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
We should tag the hadoop/hdfs config so that we can retrieve properties by there usage/application like PERFORMANCE, NAMENODE etc. Right now we don't have an option available to group or list related properties together. Grouping properties through some restricted set of Meta tags and then exposing them in Configuration class will be useful for end users.
For example, here is an config file with tags.
<configuration> <property> <name>dfs.namenode.servicerpc-bind-host</name> <value>localhost</value> <tag> REQUIRED </tag> </property> <property> <name>dfs.namenode.fs-limits.min-block-size</name> <value> 1048576 </value> <tag> PERFORMANCE,REQUIRED</tag> </property> <property> <name>dfs.namenode.logging.level</name> <value>Info</value> <tag>HDFS, DEBUG </tag> </property> <configuration>
Attachments
Attachments
- HDFS-12350.01.patch
- 20 kB
- Ajay Kumar
- HDFS-12350.02.patch
- 20 kB
- Ajay Kumar
- HDFS-12350.03.patch
- 19 kB
- Ajay Kumar
Issue Links
- is related to
-
HDFS-12529 Get source for config tags from file name
- Resolved
-
HADOOP-15007 Stabilize and document Configuration <tag> element
- Resolved
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 19s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 17m 12s | trunk passed |
+1 | compile | 18m 9s | trunk passed |
+1 | checkstyle | 0m 48s | trunk passed |
-1 | mvnsite | 0m 18s | hadoop-common in trunk failed. |
+1 | findbugs | 2m 10s | trunk passed |
+1 | javadoc | 0m 58s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 49s | the patch passed |
+1 | compile | 14m 8s | the patch passed |
+1 | javac | 14m 8s | the patch passed |
-0 | checkstyle | 0m 49s | hadoop-common-project/hadoop-common: The patch generated 7 new + 250 unchanged - 1 fixed = 257 total (was 251) |
-1 | mvnsite | 0m 19s | hadoop-common in the patch failed. |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 2m 18s | the patch passed |
+1 | javadoc | 0m 57s | the patch passed |
Other Tests | |||
-1 | unit | 9m 20s | hadoop-common in the patch failed. |
+1 | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
70m 30s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.security.TestKDiag |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 17s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 15m 17s | trunk passed |
+1 | compile | 16m 8s | trunk passed |
+1 | checkstyle | 0m 34s | trunk passed |
-1 | mvnsite | 0m 15s | hadoop-common in trunk failed. |
+1 | findbugs | 1m 52s | trunk passed |
+1 | javadoc | 0m 50s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 45s | the patch passed |
+1 | compile | 12m 7s | the patch passed |
+1 | javac | 12m 7s | the patch passed |
-0 | checkstyle | 0m 36s | hadoop-common-project/hadoop-common: The patch generated 7 new + 250 unchanged - 1 fixed = 257 total (was 251) |
-1 | mvnsite | 0m 13s | hadoop-common in the patch failed. |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 2m 5s | the patch passed |
+1 | javadoc | 0m 52s | the patch passed |
Other Tests | |||
-1 | unit | 8m 46s | hadoop-common in the patch failed. |
+1 | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
62m 3s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.security.TestShellBasedUnixGroupsMapping |
This message was automatically generated.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 16m 25s | trunk passed |
+1 | compile | 14m 50s | trunk passed |
+1 | checkstyle | 0m 37s | trunk passed |
+1 | mvnsite | 1m 29s | trunk passed |
+1 | findbugs | 1m 22s | trunk passed |
+1 | javadoc | 0m 50s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 39s | the patch passed |
+1 | compile | 10m 40s | the patch passed |
+1 | javac | 10m 40s | the patch passed |
-0 | checkstyle | 0m 38s | hadoop-common-project/hadoop-common: The patch generated 2 new + 250 unchanged - 1 fixed = 252 total (was 251) |
+1 | mvnsite | 1m 24s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 1m 33s | the patch passed |
+1 | javadoc | 0m 50s | the patch passed |
Other Tests | |||
+1 | unit | 7m 50s | hadoop-common in the patch passed. |
+1 | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
61m 1s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:14b5c93 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884491/HDFS-12350.02.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 88223ea32630 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 9992675 |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20927/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20927/testReport/ |
modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20927/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 31s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 17m 32s | trunk passed |
+1 | compile | 16m 46s | trunk passed |
+1 | checkstyle | 0m 42s | trunk passed |
+1 | mvnsite | 1m 12s | trunk passed |
+1 | findbugs | 1m 41s | trunk passed |
+1 | javadoc | 0m 55s | trunk passed |
Patch Compile Tests | |||
-1 | mvninstall | 0m 27s | hadoop-common in the patch failed. |
-1 | compile | 0m 40s | root in the patch failed. |
-1 | javac | 0m 40s | root in the patch failed. |
-0 | checkstyle | 0m 33s | hadoop-common-project/hadoop-common: The patch generated 2 new + 250 unchanged - 1 fixed = 252 total (was 251) |
-1 | mvnsite | 0m 27s | hadoop-common in the patch failed. |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
-1 | findbugs | 0m 24s | hadoop-common in the patch failed. |
-1 | javadoc | 0m 52s | hadoop-common-project_hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
Other Tests | |||
-1 | unit | 0m 25s | hadoop-common in the patch failed. |
+1 | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
44m 14s |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 1m 32s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 15m 9s | trunk passed |
+1 | compile | 19m 1s | trunk passed |
+1 | checkstyle | 0m 44s | trunk passed |
+1 | mvnsite | 1m 12s | trunk passed |
+1 | findbugs | 1m 50s | trunk passed |
+1 | javadoc | 0m 58s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 49s | the patch passed |
+1 | compile | 14m 13s | the patch passed |
+1 | javac | 14m 13s | the patch passed |
-0 | checkstyle | 0m 50s | hadoop-common-project/hadoop-common: The patch generated 3 new + 250 unchanged - 1 fixed = 253 total (was 251) |
+1 | mvnsite | 1m 19s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 1m 59s | the patch passed |
+1 | javadoc | 0m 58s | the patch passed |
Other Tests | |||
-1 | unit | 9m 1s | hadoop-common in the patch failed. |
+1 | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
71m 25s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.ha.TestZKFailoverController |
hadoop.security.TestRaceWhenRelogin |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:71bbb86 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12885685/HDFS-12350.03.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux bad0aa2a782d 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 22de944 |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/21029/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21029/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21029/testReport/ |
modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21029/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1, There are some checkstyle issues. I will fix them while committing. The test failures are not related to this.
ajayydv Thank you for the contribution, I have committed this to the trunk. Will reopen this later to port to 3.0 branch.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12811/)
HDFS-12350. Support meta tags in configs. Contributed by Ajay Kumar. (aengineer: rev a4cd101934ae5a5cad9663de872fb4ecee0d7560)
- (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
- (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
- (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/PropertyTag.java
- (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/HDFSPropertyTag.java
- (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/YarnPropertyTag.java
- (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/CorePropertyTag.java
This is a change to hadoop-common. It should have been filed and discussed there. Please don't make changes to hadoop-common in hdfs patches without some publicity. thanks
HDFS-12350This message was automatically generated.