Hadoop Common
  1. Hadoop Common
  2. HADOOP-6184

Provide a configuration dump in json format.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New Configuration.dumpConfiguration(Configuration, Writer) writes configuration parameters in the JSON format.

      Description

      Configuration dump in json format.

      1. HADOOP-6184.patch
        9 kB
        V.V.Chaitanya Krishna
      2. HADOOP-6184-1.patch
        11 kB
        V.V.Chaitanya Krishna
      3. HADOOP-6184-2.patch
        11 kB
        V.V.Chaitanya Krishna
      4. HADOOP-6184-3.patch
        11 kB
        V.V.Chaitanya Krishna
      5. HADOOP-6184-4.patch
        11 kB
        V.V.Chaitanya Krishna
      6. HADOOP-6184-5.patch
        12 kB
        V.V.Chaitanya Krishna
      7. HADOOP-6184-6.patch
        11 kB
        V.V.Chaitanya Krishna
      8. HADOOP-6184-7.patch
        12 kB
        V.V.Chaitanya Krishna
      9. HADOOP-6184-8.patch
        12 kB
        V.V.Chaitanya Krishna
      10. HADOOP-6184-9.patch
        12 kB
        V.V.Chaitanya Krishna
      11. HADOOP-6184-ydist.patch
        13 kB
        V.V.Chaitanya Krishna
      12. HADOOP-6184-ydist-1.patch
        13 kB
        V.V.Chaitanya Krishna

        Issue Links

          Activity

          Hide
          rahul k singh added a comment -

          In order to generate the dump in standard format (mapreduce - 768 )
          we need to provide a way in Configuration to generate dump in json format.

          Show
          rahul k singh added a comment - In order to generate the dump in standard format (mapreduce - 768 ) we need to provide a way in Configuration to generate dump in json format.
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch which provides method to return all the properties of configuration object as a string of Json format.

          The Json format is as follows:

          { "Properties" : [

          { "Key" : <key>, "Value" : <value>, "isFinal" : <true if the property is set to final> "resource" : <resource name which loads this property most recently> }

          ] }

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch which provides method to return all the properties of configuration object as a string of Json format. The Json format is as follows: { "Properties" : [ { "Key" : <key>, "Value" : <value>, "isFinal" : <true if the property is set to final> "resource" : <resource name which loads this property most recently> } ] }
          Hide
          Doug Cutting added a comment -

          What's the purpose of the "resource" field?

          Show
          Doug Cutting added a comment - What's the purpose of the "resource" field?
          Hide
          Sreekanth Ramakrishnan added a comment -

          in writeJson

          • reloadConfiguration() call is not required.
          • Can we make the method a static method with signature as following public static void writeJson(Configuration conf, Writer writer)
            for(Map.Entry<Object,Object> item: getProps().entrySet()) { should be changed to for(Map.Entry<Object,Object> item: conf.getProps().entrySet()) { where conf is your local variable.
          • uncomment the line dumpGenerator.flush(); while iterating over each key
          • Please dont import like import org.codehaus.jackson.*; import specific classes and interfaces.
          • Comments in testcase.
          Show
          Sreekanth Ramakrishnan added a comment - in writeJson reloadConfiguration() call is not required. Can we make the method a static method with signature as following public static void writeJson(Configuration conf, Writer writer) for(Map.Entry<Object,Object> item: getProps().entrySet()) { should be changed to for(Map.Entry<Object,Object> item: conf.getProps().entrySet()) { where conf is your local variable. uncomment the line dumpGenerator.flush(); while iterating over each key Please dont import like import org.codehaus.jackson.*; import specific classes and interfaces. Comments in testcase.
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with the above comments implemented.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with the above comments implemented.
          Hide
          V.V.Chaitanya Krishna added a comment -

          The previous patch does not handle the case when the key is first loaded by a resource file and then programatically set by the user. A new patch is uploaded which handles this issue.

          Show
          V.V.Chaitanya Krishna added a comment - The previous patch does not handle the case when the key is first loaded by a resource file and then programatically set by the user. A new patch is uploaded which handles this issue.
          Hide
          V.V.Chaitanya Krishna added a comment -

          Had an offline discussion with Sreekanth and Rahul. It was proposed to have the following things in the Json format:

          • Keys which are present in some resource and then are overriden using se method by the user should have resource displayed as "Unknown".
          • Since currently, user would be looking for resource only when json format dump is needed, storage of key-resource mapping is done only when user wants to get dump in json format. This prevents redundant storage of key-resource mapping even when it is not required.
          Show
          V.V.Chaitanya Krishna added a comment - Had an offline discussion with Sreekanth and Rahul. It was proposed to have the following things in the Json format: Keys which are present in some resource and then are overriden using se method by the user should have resource displayed as "Unknown". Since currently, user would be looking for resource only when json format dump is needed, storage of key-resource mapping is done only when user wants to get dump in json format. This prevents redundant storage of key-resource mapping even when it is not required.
          Hide
          Sreekanth Ramakrishnan added a comment -

          The code changes look fine to me.
          +1

          Show
          Sreekanth Ramakrishnan added a comment - The code changes look fine to me. +1
          Hide
          V.V.Chaitanya Krishna added a comment -

          Example for above comment:
          in mapred-config.sh:

          export HADOOP_JOBTRACKER_OPTS="-Dmapred.jobtracker.dumpconfiguration=true

          Show
          V.V.Chaitanya Krishna added a comment - Example for above comment: in mapred-config.sh: export HADOOP_JOBTRACKER_OPTS="-Dmapred.jobtracker.dumpconfiguration=true
          Hide
          V.V.Chaitanya Krishna added a comment -

          found a findbugs warning in the patch. Uploading a new patch with this warning fixed.

          Show
          V.V.Chaitanya Krishna added a comment - found a findbugs warning in the patch. Uploading a new patch with this warning fixed.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416980/HADOOP-6184-4.patch
          against trunk revision 804918.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416980/HADOOP-6184-4.patch against trunk revision 804918. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/console This message is automatically generated.
          Hide
          Hemanth Yamijala added a comment -
          • updatingResouce need not be static. Also, I think it should not be created unless the flag to store resources is set to true.
          • Leave a space between control statement (like if) and the condition
          • Rename writeJson to dumpConfiguration. Document in writeJson that it will not work if configuration is loaded from the an input stream. Update documentation to say the format in which the dump is generated. It would also be good to document the format itself as an example.
          • Why is flush called after each item written ? Can it be done only once ?
          • Should handle the case where default resources are not loaded. When a configuration object is cloned in writeJson, it loses knowledge of whether default resources were loaded with the original config being dumped or not.
          • Testcase:
            • the writers created should be closed.
            • the way we are looking up for the key from the json properties can do with some improvement. Can't we build a hashtable once the deserialization happens and do a lookup ?
          Show
          Hemanth Yamijala added a comment - updatingResouce need not be static. Also, I think it should not be created unless the flag to store resources is set to true. Leave a space between control statement (like if) and the condition Rename writeJson to dumpConfiguration. Document in writeJson that it will not work if configuration is loaded from the an input stream. Update documentation to say the format in which the dump is generated. It would also be good to document the format itself as an example. Why is flush called after each item written ? Can it be done only once ? Should handle the case where default resources are not loaded. When a configuration object is cloned in writeJson, it loses knowledge of whether default resources were loaded with the original config being dumped or not. Testcase: the writers created should be closed. the way we are looking up for the key from the json properties can do with some improvement. Can't we build a hashtable once the deserialization happens and do a lookup ?
          Hide
          V.V.Chaitanya Krishna added a comment -

          uploading patch with the above mentioned points implemented.

          Show
          V.V.Chaitanya Krishna added a comment - uploading patch with the above mentioned points implemented.
          Hide
          V.V.Chaitanya Krishna added a comment -

          The previous patch is not valid with the recent commits done. Uploading a new patch with this issue handled.

          Also, the sample parser for json format can be checked in TestConfiguration.java

          Show
          V.V.Chaitanya Krishna added a comment - The previous patch is not valid with the recent commits done. Uploading a new patch with this issue handled. Also, the sample parser for json format can be checked in TestConfiguration.java
          Hide
          Hemanth Yamijala added a comment -
          • The commit of HADOOP-6103 ensured that the loadDefaults variable is being cloned in the copy constructor. So, it can be removed from the private constructor you are introducing.

          the writers created should be closed.

          • I meant the BufferedWriter to be closed in the testcase, not the StringWriter.
          • Also for the test case related to default resources, I think we should ideally have that in a separate test case. And it should make sure that the default size of the json properties is 0. Then we can add a new resource that adds a couple of strings and makes sure those alone are dumped.
          Show
          Hemanth Yamijala added a comment - The commit of HADOOP-6103 ensured that the loadDefaults variable is being cloned in the copy constructor. So, it can be removed from the private constructor you are introducing. the writers created should be closed. I meant the BufferedWriter to be closed in the testcase, not the StringWriter. Also for the test case related to default resources, I think we should ideally have that in a separate test case. And it should make sure that the default size of the json properties is 0. Then we can add a new resource that adds a couple of strings and makes sure those alone are dumped.
          Hide
          V.V.Chaitanya Krishna added a comment -

          uploading the patch with the above points implemented.

          Show
          V.V.Chaitanya Krishna added a comment - uploading the patch with the above points implemented.
          Hide
          Hemanth Yamijala added a comment -

          Looks good. +1. Running through Hudson

          Show
          Hemanth Yamijala added a comment - Looks good. +1. Running through Hudson
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417252/HADOOP-6184-7.patch
          against trunk revision 806430.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417252/HADOOP-6184-7.patch against trunk revision 806430. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/console This message is automatically generated.
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with the correction done

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with the correction done
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417351/HADOOP-6184-8.patch
          against trunk revision 806745.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417351/HADOOP-6184-8.patch against trunk revision 806745. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/console This message is automatically generated.
          Hide
          V.V.Chaitanya Krishna added a comment -

          As pointed by Steve Loughran in MAPREDUCE-768, the expansion of configuration properties needs to be done. Uploading the patch with this being implemented.

          Show
          V.V.Chaitanya Krishna added a comment - As pointed by Steve Loughran in MAPREDUCE-768 , the expansion of configuration properties needs to be done. Uploading the patch with this being implemented.
          Hide
          Hemanth Yamijala added a comment -

          Changes look fine to me. Trying Hudson again.

          Show
          Hemanth Yamijala added a comment - Changes look fine to me. Trying Hudson again.
          Hide
          Hemanth Yamijala added a comment -

          When we complete HADOOP-6105, I can think of a useful enhancement where we can dump if some deprecated keys are being defined in the configuration.

          Show
          Hemanth Yamijala added a comment - When we complete HADOOP-6105 , I can think of a useful enhancement where we can dump if some deprecated keys are being defined in the configuration.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417439/HADOOP-6184-9.patch
          against trunk revision 806745.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417439/HADOOP-6184-9.patch against trunk revision 806745. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/console This message is automatically generated.
          Hide
          V.V.Chaitanya Krishna added a comment -

          uploading patch for Yahoo! distribution.

          Show
          V.V.Chaitanya Krishna added a comment - uploading patch for Yahoo! distribution.
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading new patch for Yahoo! distribution with the compiler error fixed.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading new patch for Yahoo! distribution with the compiler error fixed.
          Hide
          Hemanth Yamijala added a comment -

          I just committed this to trunk. Thanks, Chaitanya !

          Show
          Hemanth Yamijala added a comment - I just committed this to trunk. Thanks, Chaitanya !
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #68 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/68/)
          . Provide an API to dump Configuration in a JSON format. Contributed by V.V.Chaitanya Krishna.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #68 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/68/ ) . Provide an API to dump Configuration in a JSON format. Contributed by V.V.Chaitanya Krishna.
          Hide
          Hemanth Yamijala added a comment -

          I committed the common jar and test jar to the MAPREDUCE sub-project after verifying tests pass with the new jar.

          Show
          Hemanth Yamijala added a comment - I committed the common jar and test jar to the MAPREDUCE sub-project after verifying tests pass with the new jar.
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.

            People

            • Assignee:
              V.V.Chaitanya Krishna
              Reporter:
              rahul k singh
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development