Hadoop Common
  1. Hadoop Common
  2. HADOOP-6184

Provide a configuration dump in json format.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New Configuration.dumpConfiguration(Configuration, Writer) writes configuration parameters in the JSON format.

      Description

      Configuration dump in json format.

      1. HADOOP-6184.patch
        9 kB
        V.V.Chaitanya Krishna
      2. HADOOP-6184-1.patch
        11 kB
        V.V.Chaitanya Krishna
      3. HADOOP-6184-2.patch
        11 kB
        V.V.Chaitanya Krishna
      4. HADOOP-6184-3.patch
        11 kB
        V.V.Chaitanya Krishna
      5. HADOOP-6184-4.patch
        11 kB
        V.V.Chaitanya Krishna
      6. HADOOP-6184-5.patch
        12 kB
        V.V.Chaitanya Krishna
      7. HADOOP-6184-6.patch
        11 kB
        V.V.Chaitanya Krishna
      8. HADOOP-6184-7.patch
        12 kB
        V.V.Chaitanya Krishna
      9. HADOOP-6184-8.patch
        12 kB
        V.V.Chaitanya Krishna
      10. HADOOP-6184-9.patch
        12 kB
        V.V.Chaitanya Krishna
      11. HADOOP-6184-ydist.patch
        13 kB
        V.V.Chaitanya Krishna
      12. HADOOP-6184-ydist-1.patch
        13 kB
        V.V.Chaitanya Krishna

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          4d 23h 49m 3 Hemanth Yamijala 24/Aug/09 07:17
          Open Open Patch Available Patch Available
          8d 19h 20m 4 Hemanth Yamijala 24/Aug/09 07:17
          Patch Available Patch Available Resolved Resolved
          5h 32m 1 Hemanth Yamijala 24/Aug/09 12:50
          Resolved Resolved Closed Closed
          365d 8h 49m 1 Tom White 24/Aug/10 21:39
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Chansler made changes -
          Release Note The API dumpConfiguration(Configuration, Writer) was added to Configuration.java. Calling this API would write out all the configuration parameters and their properties (final and resource) to a specified Writer in the JSON format. The format of the output would be { "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,key2.isFinal,key2.resource}... ] }. New Configuration.dumpConfiguration(Configuration, Writer) writes configuration parameters in the JSON format.
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
          Hemanth Yamijala made changes -
          Release Note The API dumpConfiguration(Configuration, Writer) in Configuration.java writes out all the parameters and their properties (final and resource) to the given Writer. The format of the output would be { "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,key2.isFinal,key2.resource}... ] }. It does not output the parameters of the configuration object which is loaded from an input stream. The API dumpConfiguration(Configuration, Writer) was added to Configuration.java. Calling this API would write out all the configuration parameters and their properties (final and resource) to a specified Writer in the JSON format. The format of the output would be { "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,key2.isFinal,key2.resource}... ] }.
          V.V.Chaitanya Krishna made changes -
          Release Note The API dumpConfiguration(Configuration, Writer) in Configuration.java writes out all the parameters and their properties (final and resource) to the given Writer. The format of the output would be { "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,key2.isFinal,key2.resource}... ] }. It does not output the parameters of the configuration object which is loaded from an input stream.
          Hide
          Hemanth Yamijala added a comment -

          I committed the common jar and test jar to the MAPREDUCE sub-project after verifying tests pass with the new jar.

          Show
          Hemanth Yamijala added a comment - I committed the common jar and test jar to the MAPREDUCE sub-project after verifying tests pass with the new jar.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #68 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/68/)
          . Provide an API to dump Configuration in a JSON format. Contributed by V.V.Chaitanya Krishna.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #68 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/68/ ) . Provide an API to dump Configuration in a JSON format. Contributed by V.V.Chaitanya Krishna.
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12313563 ]
          Resolution Fixed [ 1 ]
          Hide
          Hemanth Yamijala added a comment -

          I just committed this to trunk. Thanks, Chaitanya !

          Show
          Hemanth Yamijala added a comment - I just committed this to trunk. Thanks, Chaitanya !
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-ydist-1.patch [ 12417472 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading new patch for Yahoo! distribution with the compiler error fixed.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading new patch for Yahoo! distribution with the compiler error fixed.
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-ydist.patch [ 12417470 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          uploading patch for Yahoo! distribution.

          Show
          V.V.Chaitanya Krishna added a comment - uploading patch for Yahoo! distribution.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417439/HADOOP-6184-9.patch
          against trunk revision 806745.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417439/HADOOP-6184-9.patch against trunk revision 806745. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/console This message is automatically generated.
          Hide
          Hemanth Yamijala added a comment -

          When we complete HADOOP-6105, I can think of a useful enhancement where we can dump if some deprecated keys are being defined in the configuration.

          Show
          Hemanth Yamijala added a comment - When we complete HADOOP-6105 , I can think of a useful enhancement where we can dump if some deprecated keys are being defined in the configuration.
          Hemanth Yamijala made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hemanth Yamijala added a comment -

          Changes look fine to me. Trying Hudson again.

          Show
          Hemanth Yamijala added a comment - Changes look fine to me. Trying Hudson again.
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-9.patch [ 12417439 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          As pointed by Steve Loughran in MAPREDUCE-768, the expansion of configuration properties needs to be done. Uploading the patch with this being implemented.

          Show
          V.V.Chaitanya Krishna added a comment - As pointed by Steve Loughran in MAPREDUCE-768 , the expansion of configuration properties needs to be done. Uploading the patch with this being implemented.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417351/HADOOP-6184-8.patch
          against trunk revision 806745.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417351/HADOOP-6184-8.patch against trunk revision 806745. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/console This message is automatically generated.
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          V.V.Chaitanya Krishna made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-8.patch [ 12417351 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with the correction done

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with the correction done
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417252/HADOOP-6184-7.patch
          against trunk revision 806430.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417252/HADOOP-6184-7.patch against trunk revision 806430. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/console This message is automatically generated.
          Hemanth Yamijala made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hemanth Yamijala added a comment -

          Looks good. +1. Running through Hudson

          Show
          Hemanth Yamijala added a comment - Looks good. +1. Running through Hudson
          Hemanth Yamijala made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-7.patch [ 12417252 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          uploading the patch with the above points implemented.

          Show
          V.V.Chaitanya Krishna added a comment - uploading the patch with the above points implemented.
          Hide
          Hemanth Yamijala added a comment -
          • The commit of HADOOP-6103 ensured that the loadDefaults variable is being cloned in the copy constructor. So, it can be removed from the private constructor you are introducing.

          the writers created should be closed.

          • I meant the BufferedWriter to be closed in the testcase, not the StringWriter.
          • Also for the test case related to default resources, I think we should ideally have that in a separate test case. And it should make sure that the default size of the json properties is 0. Then we can add a new resource that adds a couple of strings and makes sure those alone are dumped.
          Show
          Hemanth Yamijala added a comment - The commit of HADOOP-6103 ensured that the loadDefaults variable is being cloned in the copy constructor. So, it can be removed from the private constructor you are introducing. the writers created should be closed. I meant the BufferedWriter to be closed in the testcase, not the StringWriter. Also for the test case related to default resources, I think we should ideally have that in a separate test case. And it should make sure that the default size of the json properties is 0. Then we can add a new resource that adds a couple of strings and makes sure those alone are dumped.
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-6.patch [ 12417241 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          The previous patch is not valid with the recent commits done. Uploading a new patch with this issue handled.

          Also, the sample parser for json format can be checked in TestConfiguration.java

          Show
          V.V.Chaitanya Krishna added a comment - The previous patch is not valid with the recent commits done. Uploading a new patch with this issue handled. Also, the sample parser for json format can be checked in TestConfiguration.java
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-5.patch [ 12417225 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          uploading patch with the above mentioned points implemented.

          Show
          V.V.Chaitanya Krishna added a comment - uploading patch with the above mentioned points implemented.
          Hide
          Hemanth Yamijala added a comment -
          • updatingResouce need not be static. Also, I think it should not be created unless the flag to store resources is set to true.
          • Leave a space between control statement (like if) and the condition
          • Rename writeJson to dumpConfiguration. Document in writeJson that it will not work if configuration is loaded from the an input stream. Update documentation to say the format in which the dump is generated. It would also be good to document the format itself as an example.
          • Why is flush called after each item written ? Can it be done only once ?
          • Should handle the case where default resources are not loaded. When a configuration object is cloned in writeJson, it loses knowledge of whether default resources were loaded with the original config being dumped or not.
          • Testcase:
            • the writers created should be closed.
            • the way we are looking up for the key from the json properties can do with some improvement. Can't we build a hashtable once the deserialization happens and do a lookup ?
          Show
          Hemanth Yamijala added a comment - updatingResouce need not be static. Also, I think it should not be created unless the flag to store resources is set to true. Leave a space between control statement (like if) and the condition Rename writeJson to dumpConfiguration. Document in writeJson that it will not work if configuration is loaded from the an input stream. Update documentation to say the format in which the dump is generated. It would also be good to document the format itself as an example. Why is flush called after each item written ? Can it be done only once ? Should handle the case where default resources are not loaded. When a configuration object is cloned in writeJson, it loses knowledge of whether default resources were loaded with the original config being dumped or not. Testcase: the writers created should be closed. the way we are looking up for the key from the json properties can do with some improvement. Can't we build a hashtable once the deserialization happens and do a lookup ?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12416980/HADOOP-6184-4.patch
          against trunk revision 804918.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416980/HADOOP-6184-4.patch against trunk revision 804918. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/console This message is automatically generated.
          V.V.Chaitanya Krishna made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-4.patch [ 12416980 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          found a findbugs warning in the patch. Uploading a new patch with this warning fixed.

          Show
          V.V.Chaitanya Krishna added a comment - found a findbugs warning in the patch. Uploading a new patch with this warning fixed.
          V.V.Chaitanya Krishna made changes -
          Assignee V.V.Chaitanya Krishna [ chaitk ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Example for above comment:
          in mapred-config.sh:

          export HADOOP_JOBTRACKER_OPTS="-Dmapred.jobtracker.dumpconfiguration=true

          Show
          V.V.Chaitanya Krishna added a comment - Example for above comment: in mapred-config.sh: export HADOOP_JOBTRACKER_OPTS="-Dmapred.jobtracker.dumpconfiguration=true
          Hide
          Sreekanth Ramakrishnan added a comment -

          The code changes look fine to me.
          +1

          Show
          Sreekanth Ramakrishnan added a comment - The code changes look fine to me. +1
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-3.patch [ 12416178 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Had an offline discussion with Sreekanth and Rahul. It was proposed to have the following things in the Json format:

          • Keys which are present in some resource and then are overriden using se method by the user should have resource displayed as "Unknown".
          • Since currently, user would be looking for resource only when json format dump is needed, storage of key-resource mapping is done only when user wants to get dump in json format. This prevents redundant storage of key-resource mapping even when it is not required.
          Show
          V.V.Chaitanya Krishna added a comment - Had an offline discussion with Sreekanth and Rahul. It was proposed to have the following things in the Json format: Keys which are present in some resource and then are overriden using se method by the user should have resource displayed as "Unknown". Since currently, user would be looking for resource only when json format dump is needed, storage of key-resource mapping is done only when user wants to get dump in json format. This prevents redundant storage of key-resource mapping even when it is not required.
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-2.patch [ 12416170 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          The previous patch does not handle the case when the key is first loaded by a resource file and then programatically set by the user. A new patch is uploaded which handles this issue.

          Show
          V.V.Chaitanya Krishna added a comment - The previous patch does not handle the case when the key is first loaded by a resource file and then programatically set by the user. A new patch is uploaded which handles this issue.
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184-1.patch [ 12416167 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch with the above comments implemented.

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch with the above comments implemented.
          Hide
          Sreekanth Ramakrishnan added a comment -

          in writeJson

          • reloadConfiguration() call is not required.
          • Can we make the method a static method with signature as following public static void writeJson(Configuration conf, Writer writer)
            for(Map.Entry<Object,Object> item: getProps().entrySet()) { should be changed to for(Map.Entry<Object,Object> item: conf.getProps().entrySet()) { where conf is your local variable.
          • uncomment the line dumpGenerator.flush(); while iterating over each key
          • Please dont import like import org.codehaus.jackson.*; import specific classes and interfaces.
          • Comments in testcase.
          Show
          Sreekanth Ramakrishnan added a comment - in writeJson reloadConfiguration() call is not required. Can we make the method a static method with signature as following public static void writeJson(Configuration conf, Writer writer) for(Map.Entry<Object,Object> item: getProps().entrySet()) { should be changed to for(Map.Entry<Object,Object> item: conf.getProps().entrySet()) { where conf is your local variable. uncomment the line dumpGenerator.flush(); while iterating over each key Please dont import like import org.codehaus.jackson.*; import specific classes and interfaces. Comments in testcase.
          Hide
          Doug Cutting added a comment -

          What's the purpose of the "resource" field?

          Show
          Doug Cutting added a comment - What's the purpose of the "resource" field?
          V.V.Chaitanya Krishna made changes -
          Attachment HADOOP-6184.patch [ 12416044 ]
          Hide
          V.V.Chaitanya Krishna added a comment -

          Uploading patch which provides method to return all the properties of configuration object as a string of Json format.

          The Json format is as follows:

          { "Properties" : [

          { "Key" : <key>, "Value" : <value>, "isFinal" : <true if the property is set to final> "resource" : <resource name which loads this property most recently> }

          ] }

          Show
          V.V.Chaitanya Krishna added a comment - Uploading patch which provides method to return all the properties of configuration object as a string of Json format. The Json format is as follows: { "Properties" : [ { "Key" : <key>, "Value" : <value>, "isFinal" : <true if the property is set to final> "resource" : <resource name which loads this property most recently> } ] }
          rahul k singh made changes -
          Field Original Value New Value
          Link This issue blocks MAPREDUCE-768 [ MAPREDUCE-768 ]
          Hide
          rahul k singh added a comment -

          In order to generate the dump in standard format (mapreduce - 768 )
          we need to provide a way in Configuration to generate dump in json format.

          Show
          rahul k singh added a comment - In order to generate the dump in standard format (mapreduce - 768 ) we need to provide a way in Configuration to generate dump in json format.
          rahul k singh created issue -

            People

            • Assignee:
              V.V.Chaitanya Krishna
              Reporter:
              rahul k singh
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development