Hadoop Common
  1. Hadoop Common
  2. HADOOP-8567

Port conf servlet to dump running configuration to branch 1.x

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.2
    • Component/s: conf
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Users can use the conf servlet to get the server-side configuration. Users can

      1) connect to http_server_url/conf or http_server_url/conf?format=xml and get XML-based configuration description;
      2) connect to http_server_url/conf?format=json and get JSON-based configuration description.
      Show
      Users can use the conf servlet to get the server-side configuration. Users can 1) connect to http_server_url/conf or http_server_url/conf?format=xml and get XML-based configuration description; 2) connect to http_server_url/conf?format=json and get JSON-based configuration description.

      Description

      HADOOP-6408 provide conf servlet that can dump running configuration which great helps admin to trouble shooting the configuration issue. However, that patch works on branch after 0.21 only and should be backport to branch 1.x.

      1. Hadoop.8567.branch-1.001.patch
        18 kB
        Jing Zhao
      2. Hadoop.8567.branch-1.002.patch
        17 kB
        Jing Zhao
      3. Hadoop.8567.branch-1.003.patch
        18 kB
        Jing Zhao
      4. Hadoop.8567.branch-1.004.patch
        18 kB
        Jing Zhao

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          +1 for backport. This will be very useful feature on stable release.

          Show
          Suresh Srinivas added a comment - +1 for backport. This will be very useful feature on stable release.
          Hide
          Jing Zhao added a comment -

          Junping Du, Hi Junping, if you are busy, I can work on the backport and post a patch for it. Thanks!

          Show
          Jing Zhao added a comment - Junping Du , Hi Junping, if you are busy, I can work on the backport and post a patch for it. Thanks!
          Hide
          Jing Zhao added a comment -

          My current patch for the backport.

          Show
          Jing Zhao added a comment - My current patch for the backport.
          Hide
          Junping Du added a comment -

          Sorry. I owe this patch for a long time. Thanks for delivering this patch. I will help on review it.

          Show
          Junping Du added a comment - Sorry. I owe this patch for a long time. Thanks for delivering this patch. I will help on review it.
          Hide
          Suresh Srinivas added a comment -

          I do not see these changes from HADOOP-6408. Are they not relevant? Rest of the porting looks good.

          diff --git src/java/org/apache/hadoop/conf/Configuration.java src/java/org/apache/hadoop/conf/Configuration.java
          index e5a4523..7407d8b 100644
          --- src/java/org/apache/hadoop/conf/Configuration.java
          +++ src/java/org/apache/hadoop/conf/Configuration.java
          @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils;
           +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                if (other.overlay!=null) {
                  this.overlay = (Properties)other.overlay.clone();
                }
          +
          +     this.updatingResource = new HashMap<String, String>(other.updatingResource);
              }
              
               this.finalParameters = new HashSet<String>(other.finalParameters);
          @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
               if (!isDeprecated(name)) {
                 getOverlay().setProperty(name, value);
                 getProps().setProperty(name, value);
          +      updatingResource.put(name, UNKNOWN_RESOURCE);
               }
               else {
                 DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name);
          @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                 if (finalParameters.contains(oldKey)) {
                   finalParameters.remove(oldKey);
                 }
          -      if (storeResource) {
          -        updatingResource.remove(oldKey);
          -      }
          +      updatingResource.remove(oldKey);
               }
             }
             
          @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                   continue;
                 }
                 properties.setProperty(key, value);
          -      if (storeResource) {
          -        updatingResource.put(key, updatingResource.get(attr));
          -      }
          +      updatingResource.put(key, updatingResource.get(attr));
                 if (finalParameter) {
                   finalParameters.add(key);
                 }
          
          Show
          Suresh Srinivas added a comment - I do not see these changes from HADOOP-6408 . Are they not relevant? Rest of the porting looks good. diff --git src/java/org/apache/hadoop/conf/Configuration.java src/java/org/apache/hadoop/conf/Configuration.java index e5a4523..7407d8b 100644 --- src/java/org/apache/hadoop/conf/Configuration.java +++ src/java/org/apache/hadoop/conf/Configuration.java @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils; +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (other.overlay!=null) { this.overlay = (Properties)other.overlay.clone(); } + + this.updatingResource = new HashMap<String, String>(other.updatingResource); } this.finalParameters = new HashSet<String>(other.finalParameters); @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (!isDeprecated(name)) { getOverlay().setProperty(name, value); getProps().setProperty(name, value); + updatingResource.put(name, UNKNOWN_RESOURCE); } else { DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name); @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (finalParameters.contains(oldKey)) { finalParameters.remove(oldKey); } - if (storeResource) { - updatingResource.remove(oldKey); - } + updatingResource.remove(oldKey); } } @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, continue; } properties.setProperty(key, value); - if (storeResource) { - updatingResource.put(key, updatingResource.get(attr)); - } + updatingResource.put(key, updatingResource.get(attr)); if (finalParameter) { finalParameters.add(key); }
          Hide
          Jing Zhao added a comment -

          Thanks for the review Suresh!

          @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils;
           +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                if (other.overlay!=null) {
                  this.overlay = (Properties)other.overlay.clone();
                }
          +
          +     this.updatingResource = new HashMap<String, String>(other.updatingResource);
              }
          

          This has been addressed in the 002 patch.

          @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
               if (!isDeprecated(name)) {
                 getOverlay().setProperty(name, value);
                 getProps().setProperty(name, value);
          +      updatingResource.put(name, UNKNOWN_RESOURCE);
               }
               else {
                 DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name);
          

          For this, I think I should add "updatingResource.put(name, UNKNOWN_RESOURCE);" to Configuration#set(). Will address that in the new patch.

          @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                 if (finalParameters.contains(oldKey)) {
                   finalParameters.remove(oldKey);
                 }
          -      if (storeResource) {
          -        updatingResource.remove(oldKey);
          -      }
          +      updatingResource.remove(oldKey);
               }
             }
          

          For this part, I did not find similar code (should be some code corresponding to remove) in both branch-1 and current trunk.

          @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
                   continue;
                 }
                 properties.setProperty(key, value);
          -      if (storeResource) {
          -        updatingResource.put(key, updatingResource.get(attr));
          -      }
          +      updatingResource.put(key, updatingResource.get(attr));
                 if (finalParameter) {
                   finalParameters.add(key);
                 }
          

          The similar change happened in Configuration#loadResource in 002 patch.

          Show
          Jing Zhao added a comment - Thanks for the review Suresh! @@ -68,6 +70,7 @@ import org.apache.hadoop.util.ReflectionUtils; +410,8 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (other.overlay!=null) { this.overlay = (Properties)other.overlay.clone(); } + + this.updatingResource = new HashMap<String, String>(other.updatingResource); } This has been addressed in the 002 patch. @@ -604,6 +593,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (!isDeprecated(name)) { getOverlay().setProperty(name, value); getProps().setProperty(name, value); + updatingResource.put(name, UNKNOWN_RESOURCE); } else { DeprecatedKeyInfo keyInfo = deprecatedKeyMap.get(name); For this, I think I should add "updatingResource.put(name, UNKNOWN_RESOURCE);" to Configuration#set(). Will address that in the new patch. @@ -1438,9 +1426,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, if (finalParameters.contains(oldKey)) { finalParameters.remove(oldKey); } - if (storeResource) { - updatingResource.remove(oldKey); - } + updatingResource.remove(oldKey); } } For this part, I did not find similar code (should be some code corresponding to remove) in both branch-1 and current trunk. @@ -1464,9 +1450,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>, continue; } properties.setProperty(key, value); - if (storeResource) { - updatingResource.put(key, updatingResource.get(attr)); - } + updatingResource.put(key, updatingResource.get(attr)); if (finalParameter) { finalParameters.add(key); } The similar change happened in Configuration#loadResource in 002 patch.
          Hide
          Jing Zhao added a comment -

          New patch addressing Suresh's comments.

          Show
          Jing Zhao added a comment - New patch addressing Suresh's comments.
          Hide
          Suresh Srinivas added a comment -

          +1 for the latest patch. I committed it to branch-1.

          Show
          Suresh Srinivas added a comment - +1 for the latest patch. I committed it to branch-1.
          Hide
          Suresh Srinivas added a comment -

          Previous patch did not add the servlet to http server. I reverted the change. The new patch looks good.

          Jing can you please make sure servlet is added by manually testing it as well?

          Show
          Suresh Srinivas added a comment - Previous patch did not add the servlet to http server. I reverted the change. The new patch looks good. Jing can you please make sure servlet is added by manually testing it as well?
          Hide
          Jing Zhao added a comment -

          Yes. I've manually tested the new patch and could get the configuration through the conf servlet.

          Show
          Jing Zhao added a comment - Yes. I've manually tested the new patch and could get the configuration through the conf servlet.
          Hide
          Suresh Srinivas added a comment -

          Reopening to commit the issue.

          Show
          Suresh Srinivas added a comment - Reopening to commit the issue.
          Hide
          Suresh Srinivas added a comment -

          Committed the patch to branch-1. Thank you Jing.

          Show
          Suresh Srinivas added a comment - Committed the patch to branch-1. Thank you Jing.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Merged to branch-1.1.

          Show
          Tsz Wo Nicholas Sze added a comment - Merged to branch-1.1.
          Hide
          Matt Foley added a comment -

          Closed upon successful release of 1.1.2.

          Show
          Matt Foley added a comment - Closed upon successful release of 1.1.2.

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development