Hadoop Common
  1. Hadoop Common
  2. HADOOP-8567

Port conf servlet to dump running configuration to branch 1.x

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.2
    • Component/s: conf
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Users can use the conf servlet to get the server-side configuration. Users can

      1) connect to http_server_url/conf or http_server_url/conf?format=xml and get XML-based configuration description;
      2) connect to http_server_url/conf?format=json and get JSON-based configuration description.
      Show
      Users can use the conf servlet to get the server-side configuration. Users can 1) connect to http_server_url/conf or http_server_url/conf?format=xml and get XML-based configuration description; 2) connect to http_server_url/conf?format=json and get JSON-based configuration description.

      Description

      HADOOP-6408 provide conf servlet that can dump running configuration which great helps admin to trouble shooting the configuration issue. However, that patch works on branch after 0.21 only and should be backport to branch 1.x.

      1. Hadoop.8567.branch-1.004.patch
        18 kB
        Jing Zhao
      2. Hadoop.8567.branch-1.003.patch
        18 kB
        Jing Zhao
      3. Hadoop.8567.branch-1.002.patch
        17 kB
        Jing Zhao
      4. Hadoop.8567.branch-1.001.patch
        18 kB
        Jing Zhao

        Issue Links

          Activity

          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          Closed upon successful release of 1.1.2.

          Show
          Matt Foley added a comment - Closed upon successful release of 1.1.2.
          Matt Foley made changes -
          Target Version/s 1.2.0 [ 12321659 ] 1.1.2 [ 12323596 ]
          Tsz Wo Nicholas Sze made changes -
          Fix Version/s 1.1.2 [ 12323596 ]
          Fix Version/s 1.2.0 [ 12321659 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Merged to branch-1.1.

          Show
          Tsz Wo Nicholas Sze added a comment - Merged to branch-1.1.
          Suresh Srinivas made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 1.2.0 [ 12321659 ]
          Resolution Fixed [ 1 ]
          Hide
          Suresh Srinivas added a comment -

          Committed the patch to branch-1. Thank you Jing.

          Show
          Suresh Srinivas added a comment - Committed the patch to branch-1. Thank you Jing.
          Suresh Srinivas made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Suresh Srinivas added a comment -

          Reopening to commit the issue.

          Show
          Suresh Srinivas added a comment - Reopening to commit the issue.
          Jing Zhao made changes -
          Release Note Users can use the conf servlet to get the server-side configuration. Users can

          1) connect to http_server_url/conf or http_server_url/conf?format=xml and get XML-based configuration description;
          2) connect to http_server_url/conf?format=json and get JSON-based configuration description.
          Hide
          Jing Zhao added a comment -

          Yes. I've manually tested the new patch and could get the configuration through the conf servlet.

          Show
          Jing Zhao added a comment - Yes. I've manually tested the new patch and could get the configuration through the conf servlet.
          Hide
          Suresh Srinivas added a comment -

          Previous patch did not add the servlet to http server. I reverted the change. The new patch looks good.

          Jing can you please make sure servlet is added by manually testing it as well?

          Show
          Suresh Srinivas added a comment - Previous patch did not add the servlet to http server. I reverted the change. The new patch looks good. Jing can you please make sure servlet is added by manually testing it as well?
          Jing Zhao made changes -
          Attachment Hadoop.8567.branch-1.004.patch [ 12551038 ]
          Suresh Srinivas made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Suresh Srinivas made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Suresh Srinivas added a comment -

          +1 for the latest patch. I committed it to branch-1.

          Show
          Suresh Srinivas added a comment - +1 for the latest patch. I committed it to branch-1.
          Suresh Srinivas made changes -
          Summary Backport conf servlet with dump running configuration to branch 1.x Port conf servlet to dump running configuration to branch 1.x
          Jing Zhao made changes -
          Attachment Hadoop.8567.branch-1.003.patch [ 12550996 ]
          Hide
          Jing Zhao added a comment -

          New patch addressing Suresh's comments.

          Show
          Jing Zhao added a comment - New patch addressing Suresh's comments.
          Hide
          Jing Zhao added a comment -

          Thanks for the review Suresh!

          @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils;
           +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                if (other.overlay!=null) {
                  this.overlay = (Properties)other.overlay.clone();
                }
          +
          +     this.updatingResource = new HashMap<String, String>(other.updatingResource);
              }
          

          This has been addressed in the 002 patch.

          @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
               if (!isDeprecated(name)) {
                 getOverlay().setProperty(name, value);
                 getProps().setProperty(name, value);
          +      updatingResource.put(name, UNKNOWN_RESOURCE);
               }
               else {
                 DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name);
          

          For this, I think I should add "updatingResource.put(name, UNKNOWN_RESOURCE);" to Configuration#set(). Will address that in the new patch.

          @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                 if (finalParameters.contains(oldKey)) {
                   finalParameters.remove(oldKey);
                 }
          -      if (storeResource) {
          -        updatingResource.remove(oldKey);
          -      }
          +      updatingResource.remove(oldKey);
               }
             }
          

          For this part, I did not find similar code (should be some code corresponding to remove) in both branch-1 and current trunk.

          @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                   continue;
                 }
                 properties.setProperty(key, value);
          -      if (storeResource) {
          -        updatingResource.put(key, updatingResource.get(attr));
          -      }
          +      updatingResource.put(key, updatingResource.get(attr));
                 if (finalParameter) {
                   finalParameters.add(key);
                 }
          

          The similar change happened in Configuration#loadResource in 002 patch.

          Show
          Jing Zhao added a comment - Thanks for the review Suresh! @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils; +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (other.overlay!=null) { this.overlay = (Properties)other.overlay.clone(); } + + this.updatingResource = new HashMap<String, String>(other.updatingResource); } This has been addressed in the 002 patch. @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (!isDeprecated(name)) { getOverlay().setProperty(name, value); getProps().setProperty(name, value); + updatingResource.put(name, UNKNOWN_RESOURCE); } else { DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name); For this, I think I should add "updatingResource.put(name, UNKNOWN_RESOURCE);" to Configuration#set(). Will address that in the new patch. @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (finalParameters.contains(oldKey)) { finalParameters.remove(oldKey); } - if (storeResource) { - updatingResource.remove(oldKey); - } + updatingResource.remove(oldKey); } } For this part, I did not find similar code (should be some code corresponding to remove) in both branch-1 and current trunk. @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, continue; } properties.setProperty(key, value); - if (storeResource) { - updatingResource.put(key, updatingResource.get(attr)); - } + updatingResource.put(key, updatingResource.get(attr)); if (finalParameter) { finalParameters.add(key); } The similar change happened in Configuration#loadResource in 002 patch.
          Hide
          Suresh Srinivas added a comment -

          I do not see these changes from HADOOP-6408. Are they not relevant? Rest of the porting looks good.

          diff --git src/java/org/apache/hadoop/conf/Configuration.java src/java/org/apache/hadoop/conf/Configuration.java
          index e5a4523..7407d8b 100644
          --- src/java/org/apache/hadoop/conf/Configuration.java
          +++ src/java/org/apache/hadoop/conf/Configuration.java
          @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils;
           +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                if (other.overlay!=null) {
                  this.overlay = (Properties)other.overlay.clone();
                }
          +
          +     this.updatingResource = new HashMap<String, String>(other.updatingResource);
              }
              
               this.finalParameters = new HashSet<String>(other.finalParameters);
          @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
               if (!isDeprecated(name)) {
                 getOverlay().setProperty(name, value);
                 getProps().setProperty(name, value);
          +      updatingResource.put(name, UNKNOWN_RESOURCE);
               }
               else {
                 DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name);
          @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                 if (finalParameters.contains(oldKey)) {
                   finalParameters.remove(oldKey);
                 }
          -      if (storeResource) {
          -        updatingResource.remove(oldKey);
          -      }
          +      updatingResource.remove(oldKey);
               }
             }
             
          @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                   continue;
                 }
                 properties.setProperty(key, value);
          -      if (storeResource) {
          -        updatingResource.put(key, updatingResource.get(attr));
          -      }
          +      updatingResource.put(key, updatingResource.get(attr));
                 if (finalParameter) {
                   finalParameters.add(key);
                 }
          
          Show
          Suresh Srinivas added a comment - I do not see these changes from HADOOP-6408 . Are they not relevant? Rest of the porting looks good. diff --git src/java/org/apache/hadoop/conf/Configuration.java src/java/org/apache/hadoop/conf/Configuration.java index e5a4523..7407d8b 100644 --- src/java/org/apache/hadoop/conf/Configuration.java +++ src/java/org/apache/hadoop/conf/Configuration.java @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils; +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (other.overlay!=null) { this.overlay = (Properties)other.overlay.clone(); } + + this.updatingResource = new HashMap<String, String>(other.updatingResource); } this.finalParameters = new HashSet<String>(other.finalParameters); @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (!isDeprecated(name)) { getOverlay().setProperty(name, value); getProps().setProperty(name, value); + updatingResource.put(name, UNKNOWN_RESOURCE); } else { DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name); @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (finalParameters.contains(oldKey)) { finalParameters.remove(oldKey); } - if (storeResource) { - updatingResource.remove(oldKey); - } + updatingResource.remove(oldKey); } } @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, continue; } properties.setProperty(key, value); - if (storeResource) { - updatingResource.put(key, updatingResource.get(attr)); - } + updatingResource.put(key, updatingResource.get(attr)); if (finalParameter) { finalParameters.add(key); }
          Jing Zhao made changes -
          Attachment Hadoop.8567.branch-1.002.patch [ 12549405 ]
          Junping Du made changes -
          Assignee Junping Du [ djp ] Jing Zhao [ jingzhao ]
          Hide
          Junping Du added a comment -

          Sorry. I owe this patch for a long time. Thanks for delivering this patch. I will help on review it.

          Show
          Junping Du added a comment - Sorry. I owe this patch for a long time. Thanks for delivering this patch. I will help on review it.
          Jing Zhao made changes -
          Attachment Hadoop.8567.branch-1.001.patch [ 12549256 ]
          Hide
          Jing Zhao added a comment -

          My current patch for the backport.

          Show
          Jing Zhao added a comment - My current patch for the backport.
          Hide
          Jing Zhao added a comment -

          Junping Du, Hi Junping, if you are busy, I can work on the backport and post a patch for it. Thanks!

          Show
          Jing Zhao added a comment - Junping Du , Hi Junping, if you are busy, I can work on the backport and post a patch for it. Thanks!
          Harsh J made changes -
          Affects Version/s 1.0.0 [ 12318244 ]
          Affects Version/s 1.0.3 [ 12320248 ]
          Harsh J made changes -
          Fix Version/s 0.21.1 [ 12315270 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Target Version/s 1.2.0 [ 12321659 ]
          Harsh J made changes -
          Field Original Value New Value
          Link This issue is superceded by HADOOP-6408 [ HADOOP-6408 ]
          Hide
          Suresh Srinivas added a comment -

          +1 for backport. This will be very useful feature on stable release.

          Show
          Suresh Srinivas added a comment - +1 for backport. This will be very useful feature on stable release.
          Junping Du created issue -

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development