Hadoop Common
  1. Hadoop Common
  2. HADOOP-8524

Allow users to get source of a Configuration parameter

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When we load the various XMLs via the Configuration class, the source of the XML file (filename) is usually kept in the Configuration class but not exposed programmatically. It is presently exposed as comments such as "Loaded from mapred-site.xml" in the XML dump/serialization but can't be accessed otherwise (Via the Configuration API).

      For debugging/etc. purposes, it may be useful to expose this safely (such as an API for "where did this property come from?" queries for a specific property, via an API.

      1. HADOOP-8524.patch
        3 kB
        Harsh J
      2. HADOOP-8524.patch
        3 kB
        Harsh J
      3. HADOOP-8524.patch
        3 kB
        Harsh J

        Issue Links

          Activity

          Harsh J created issue -
          Harsh J made changes -
          Field Original Value New Value
          Assignee Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Attachment HADOOP-8524.patch [ 12532913 ]
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Harsh J made changes -
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Target Version/s 2.0.1-alpha, 3.0.0 [ 12321441, 12320357 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12532913/HADOOP-8524.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.fs.viewfs.TestViewFsTrash

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1132//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1132//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1132//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532913/HADOOP-8524.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1132//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1132//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1132//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          New patch that should fix the findbugs sync issue (wasn't sync'd over the internal DSes).

          The javadoc and the failing test warnings are unrelated to this one.

          Show
          Harsh J added a comment - New patch that should fix the findbugs sync issue (wasn't sync'd over the internal DSes). The javadoc and the failing test warnings are unrelated to this one.
          Harsh J made changes -
          Attachment HADOOP-8524.patch [ 12532927 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12532927/HADOOP-8524.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1134//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1134//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532927/HADOOP-8524.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1134//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1134//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          The code itself looks good to me. The only issue I have has nothing to do with your code. In the past I have talked with others about improving the debuggability/traceability of these values. For Map/Reduce jobs the config is produced and then written out to job.xml. After that the config will only look like all of the settings came from job.xml. We lose traceability to the original configurations. This is even worse for the job.xml that gets sent to the job history server, because it is written out yet again after being read in by the AM, so even the comments in there only say job.xml.

          I don't really expect you to fix this in your patch, unless you really want to. It would just be nice to have the API marked as @Unstable so that if I ever do get around to putting in better traceability we don't have to change this API. If you are OK with the idea and marking it unstable I will file a separate JIRA for actually adding in the traceability.

          Show
          Robert Joseph Evans added a comment - The code itself looks good to me. The only issue I have has nothing to do with your code. In the past I have talked with others about improving the debuggability/traceability of these values. For Map/Reduce jobs the config is produced and then written out to job.xml. After that the config will only look like all of the settings came from job.xml. We lose traceability to the original configurations. This is even worse for the job.xml that gets sent to the job history server, because it is written out yet again after being read in by the AM, so even the comments in there only say job.xml. I don't really expect you to fix this in your patch, unless you really want to. It would just be nice to have the API marked as @Unstable so that if I ever do get around to putting in better traceability we don't have to change this API. If you are OK with the idea and marking it unstable I will file a separate JIRA for actually adding in the traceability.
          Hide
          Harsh J added a comment -

          Hey,

          That makes complete sense and I am also perfectly fine with the idea of marking this as unstable. I do love the idea of proper traceability and can work on it if no one's begun yet (or if you're not gonna be driving it yet). Do file one and chalk it out.

          This request arose out of Oozie configs however at my end, not purely MR but yes the traceability will help there too.

          I've attached a patch to mark the API unstable.

          Thanks for the comment and for taking a look quickly!

          Show
          Harsh J added a comment - Hey, That makes complete sense and I am also perfectly fine with the idea of marking this as unstable. I do love the idea of proper traceability and can work on it if no one's begun yet (or if you're not gonna be driving it yet). Do file one and chalk it out. This request arose out of Oozie configs however at my end, not purely MR but yes the traceability will help there too. I've attached a patch to mark the API unstable. Thanks for the comment and for taking a look quickly!
          Harsh J made changes -
          Attachment HADOOP-8524.patch [ 12532939 ]
          Hide
          Robert Joseph Evans added a comment -

          Great I just filed HADOOP-8525 for this. If you want to work on it go right ahead, if not I will try to throw together a quick patch sometime this week.

          and +1 for this patch. Looks good to me.

          Show
          Robert Joseph Evans added a comment - Great I just filed HADOOP-8525 for this. If you want to work on it go right ahead, if not I will try to throw together a quick patch sometime this week. and +1 for this patch. Looks good to me.
          Harsh J made changes -
          Link This issue is superceded by HADOOP-8525 [ HADOOP-8525 ]
          Hide
          Harsh J added a comment -

          Thanks Robert. Committed to branch-2 and trunk.

          Show
          Harsh J added a comment - Thanks Robert. Committed to branch-2 and trunk.
          Harsh J made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.1-alpha [ 12321441 ]
          Resolution Fixed [ 1 ]
          Harsh J made changes -
          Target Version/s 2.0.1-alpha, 3.0.0 [ 12321441, 12320357 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2378 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2378/)
          HADOOP-8524. Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2378 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2378/ ) HADOOP-8524 . Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2448 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2448/)
          HADOOP-8524. Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2448 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2448/ ) HADOOP-8524 . Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2397 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2397/)
          HADOOP-8524. Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2397 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2397/ ) HADOOP-8524 . Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1084 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1084/)
          HADOOP-8524. Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1084 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1084/ ) HADOOP-8524 . Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1117 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1117/)
          HADOOP-8524. Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1117 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1117/ ) HADOOP-8524 . Allow users to get source of a Configuration parameter. (harsh) (Revision 1352689) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352689 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322473 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development