Hadoop Common
  1. Hadoop Common
  2. HADOOP-7001

Allow configuration changes without restarting configured nodes

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:

      interface ChangeableConfigured extends Configured

      { void changeConfiguration(Configuration newConf) throws ConfigurationChangeException; }

      The contract of changeConfiguration is as follows:
      The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.

      It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

      1. HADOOP-7001.5.patch
        31 kB
        Patrick Kling
      2. HADOOP-7001.4.patch
        31 kB
        Patrick Kling
      3. HADOOP-7001.3.patch
        37 kB
        Patrick Kling
      4. HADOOP-7001.2.patch
        32 kB
        Patrick Kling
      5. HADOOP-7001.patch
        32 kB
        Patrick Kling
      6. reconfigurable.patch
        9 kB
        Patrick Kling

        Issue Links

          Activity

          Hide
          Aaron T. Myers added a comment -

          +1 - this would be extremely useful.

          May I suggest, though, that we change the name of the interface to "Reconfigurable" ?

          Show
          Aaron T. Myers added a comment - +1 - this would be extremely useful. May I suggest, though, that we change the name of the interface to "Reconfigurable" ?
          Hide
          Patrick Kling added a comment -

          I agree, Reconfigurable sounds better. Also, we would be extending the interface Configurable (not the class Configured, I got confused there). I also wanted to point out that Reconfigurable would introduce the new method changeConf in addition to the method setConf from Configurable because there is a difference in semantics:
          changeConf would make sure that all changes immediately affect the behaviour of the node (or failing that an exception would be thrown), whereas setConf as it is used now does not give any such guarantee (since some configuration values might be cached in other variables and some behaviour might not be changeable without a restart).

          Show
          Patrick Kling added a comment - I agree, Reconfigurable sounds better. Also, we would be extending the interface Configurable (not the class Configured, I got confused there). I also wanted to point out that Reconfigurable would introduce the new method changeConf in addition to the method setConf from Configurable because there is a difference in semantics: changeConf would make sure that all changes immediately affect the behaviour of the node (or failing that an exception would be thrown), whereas setConf as it is used now does not give any such guarantee (since some configuration values might be cached in other variables and some behaviour might not be changeable without a restart).
          Hide
          Konstantin Boudnik added a comment -

          Also, this Herriot system testing framework will be highly benefited from such a change. +1 on the idea.

          Show
          Konstantin Boudnik added a comment - Also, this Herriot system testing framework will be highly benefited from such a change. +1 on the idea.
          Hide
          dhruba borthakur added a comment -

          It would be nice if you can paste the javadocs for this new interface and the method(s) in it. This will help understand the new API.

          Show
          dhruba borthakur added a comment - It would be nice if you can paste the javadocs for this new interface and the method(s) in it. This will help understand the new API.
          Hide
          Patrick Kling added a comment -

          org.apache.hadoop.conf
          Interface Reconfigurable

          All Superinterfaces:
          Configurable
          ___________________________________________________________________________________________________________________________________________

          public interface Reconfigurable

          extends Configurable
          Something whose Configuration can be changed at run time.
          ___________________________________________________________________________________________________________________________________________

          Method Summary
          void changeConf(Configuration conf)
          changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is
          thrown and no changes are made to the current configuration

          Methods inherited from interface org.apache.hadoop.conf.Configurable
          getConf, setConf

          Method Detail

          changeConf

          void changeConf(Configuration conf)
          throws ConfigurationChangeException

          changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is thrown
          and no changes are made to the current configuration

          Throws:
          ConfigurationChangeException

          Class ConfigurationChangeException

          java.lang.Object
          extended by java.lang.Throwable
          extended by java.lang.Exception
          extended by org.apache.hadoop.conf.ConfigurationChangeException

          All Implemented Interfaces:
          Serializable
          ___________________________________________________________________________________________________________________________________________

          public class ConfigurationChangeException

          extends Exception

          exception indicating that configuration property cannot be changed at run time

          See Also:
          Serialized Form
          ___________________________________________________________________________________________________________________________________________

          Constructor Summary
          ConfigurationChangeException(String property)
          Creates a new instance of ConfigurationChangeException
          ConfigurationChangeException(String property, String newVal)
          Creates a new instance of ConfigurationChangeException
          ConfigurationChangeException(String property, String newVal, String oldVal)
          Creates a new instance of ConfigurationChangeException

          Method Summary
          String getMessage()
          gets message describing exception
          String getNewValue()
          gets value to which property was supposed to be changed
          String getOldValue()
          gets old value of property that cannot be changed
          String getProperty()
          gets property that cannot be changed

          Methods inherited from class java.lang.Throwable
          fillInStackTrace, getCause, getLocalizedMessage, getStackTrace, initCause, printStackTrace, printStackTrace, printStackTrace, setStackTrace, toString

          Methods inherited from class java.lang.Object
          clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

          Constructor Detail

          ConfigurationChangeException

          public ConfigurationChangeException(String property,
          String newVal,
          String oldVal)

          Creates a new instance of ConfigurationChangeException
          ___________________________________________________________________________________________________________________________________________

          ConfigurationChangeException

          public ConfigurationChangeException(String property,
          String newVal)

          Creates a new instance of ConfigurationChangeException
          ___________________________________________________________________________________________________________________________________________

          ConfigurationChangeException

          public ConfigurationChangeException(String property)

          Creates a new instance of ConfigurationChangeException

          getProperty

          public String getProperty()

          gets property that cannot be changed
          ___________________________________________________________________________________________________________________________________________

          getNewValue

          public String getNewValue()

          gets value to which property was supposed to be changed
          ___________________________________________________________________________________________________________________________________________

          getOldValue

          public String getOldValue()

          gets old value of property that cannot be changed
          ___________________________________________________________________________________________________________________________________________

          getMessage

          public String getMessage()

          gets message describing exception

          Overrides:
          getMessage in class Throwable

          Show
          Patrick Kling added a comment - org.apache.hadoop.conf Interface Reconfigurable All Superinterfaces: Configurable ___________________________________________________________________________________________________________________________________________ public interface Reconfigurable extends Configurable Something whose Configuration can be changed at run time. ___________________________________________________________________________________________________________________________________________ Method Summary void changeConf(Configuration conf) changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is thrown and no changes are made to the current configuration Methods inherited from interface org.apache.hadoop.conf.Configurable getConf, setConf Method Detail changeConf void changeConf(Configuration conf) throws ConfigurationChangeException changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is thrown and no changes are made to the current configuration Throws: ConfigurationChangeException Class ConfigurationChangeException java.lang.Object extended by java.lang.Throwable extended by java.lang.Exception extended by org.apache.hadoop.conf.ConfigurationChangeException All Implemented Interfaces: Serializable ___________________________________________________________________________________________________________________________________________ public class ConfigurationChangeException extends Exception exception indicating that configuration property cannot be changed at run time See Also: Serialized Form ___________________________________________________________________________________________________________________________________________ Constructor Summary ConfigurationChangeException(String property) Creates a new instance of ConfigurationChangeException ConfigurationChangeException(String property, String newVal) Creates a new instance of ConfigurationChangeException ConfigurationChangeException(String property, String newVal, String oldVal) Creates a new instance of ConfigurationChangeException Method Summary String getMessage() gets message describing exception String getNewValue() gets value to which property was supposed to be changed String getOldValue() gets old value of property that cannot be changed String getProperty() gets property that cannot be changed Methods inherited from class java.lang.Throwable fillInStackTrace, getCause, getLocalizedMessage, getStackTrace, initCause, printStackTrace, printStackTrace, printStackTrace, setStackTrace, toString Methods inherited from class java.lang.Object clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait Constructor Detail ConfigurationChangeException public ConfigurationChangeException(String property, String newVal, String oldVal) Creates a new instance of ConfigurationChangeException ___________________________________________________________________________________________________________________________________________ ConfigurationChangeException public ConfigurationChangeException(String property, String newVal) Creates a new instance of ConfigurationChangeException ___________________________________________________________________________________________________________________________________________ ConfigurationChangeException public ConfigurationChangeException(String property) Creates a new instance of ConfigurationChangeException getProperty public String getProperty() gets property that cannot be changed ___________________________________________________________________________________________________________________________________________ getNewValue public String getNewValue() gets value to which property was supposed to be changed ___________________________________________________________________________________________________________________________________________ getOldValue public String getOldValue() gets old value of property that cannot be changed ___________________________________________________________________________________________________________________________________________ getMessage public String getMessage() gets message describing exception Overrides: getMessage in class Throwable
          Hide
          Philip Zeyliger added a comment -

          I might be grumpy, but I think the right way to deal with configuration changes in a distributed, fault-tolerant system is to just restart the daemon entirely. We already have to deal with the daemon suddenly crashing and that not affecting the system too much, so I'm wary of extra complexity of yet another process. In practice, many configuration variables are loaded at start time and then stored as statics: stuff like threadpool sizes, for example. Not to mention that Configuration objects get copied along, so it's hard to make sure that a configuration change propagates to all possible children.

          I'll point out that the namenode and the jobtracker's fair scheduler already have mechanisms for dynamic configuration changes. In namenode, that's -refreshNodes. In the jt, I think the fair scheduler re-reads its XML configuration file on occasion. Both of these make a lot of sense: these are specific endpoints for managing specific data, and the semantics of those changes are easy to understand.

          – Philip

          Show
          Philip Zeyliger added a comment - I might be grumpy, but I think the right way to deal with configuration changes in a distributed, fault-tolerant system is to just restart the daemon entirely. We already have to deal with the daemon suddenly crashing and that not affecting the system too much, so I'm wary of extra complexity of yet another process. In practice, many configuration variables are loaded at start time and then stored as statics: stuff like threadpool sizes, for example. Not to mention that Configuration objects get copied along, so it's hard to make sure that a configuration change propagates to all possible children. I'll point out that the namenode and the jobtracker's fair scheduler already have mechanisms for dynamic configuration changes. In namenode, that's -refreshNodes. In the jt, I think the fair scheduler re-reads its XML configuration file on occasion. Both of these make a lot of sense: these are specific endpoints for managing specific data, and the semantics of those changes are easy to understand. – Philip
          Hide
          dhruba borthakur added a comment -

          But a simple thing like increasing the number of handler threads in a server require the downtime of the server. If we can make the namenode implement Reconfigurable, then we will have the ability to change a few select of parameters dynamically, isn't it?

          Show
          dhruba borthakur added a comment - But a simple thing like increasing the number of handler threads in a server require the downtime of the server. If we can make the namenode implement Reconfigurable, then we will have the ability to change a few select of parameters dynamically, isn't it?
          Hide
          Patrick Kling added a comment -

          I absolutely agree that there will likely be many configuration options that cannot be changed at run time and implementing Reconfigurable does not require that we can make arbitrary changes. For the configuration changes that are possible at run time (such as the existing examples mentioned by Philip), it would be nice to have a consistent mechanism. The interface could also make it easier to propagate configuration changes (by calling changeConf on objects to which the current configuration was propagated).

          This should be a bit easier to read than the JavaDoc I posted before:

          package org.apache.hadoop.conf;
          
          /**                                                                                                                                                            
           * Something whose {@link Configuration} can be changed at run time.                                                                                           
           */
          public interface Reconfigurable extends Configurable {
          
            /**                                                                                                                                                          
             * Change the configuration of this object.                                                                                                                  
             *                                                                                                                                                           
             * Change the configuration of this object  to the configuration                                                                                             
             * passed as conf.                                                                                                                                           
             * If it is not possible to change a configuration option,                                                                                                    
             * a {@link ConfigurationChangeException} is thrown                                                                                                          
             * and no changes are made to the current configuration                                                                                                      
             */
            void changeConf(Configuration conf) throws ConfigurationChangeException;
          
          }
          
          package org.apache.hadoop.conf;
          
          
          /**                                                                                                                                                            
           * Exception indicating that configuration property cannot be changed                                                                                          
           * at run time.                                                                                                                                                
           */
          public class ConfigurationChangeException extends Exception {
          
            private String property;
            private String newVal;
            private String oldVal;
          
          
            /**                                                                                                                                                          
             * Construct the exception message.                                                                                                                          
             */
            private static String constructMessage(String property,
                                                   String newVal, String oldVal) {
              String message = "Could not change property " + property;
              if (oldVal != null) {
                message += " from " + oldVal;
              }
              if (newVal != null) {
                message += " to " + newVal;
              }
              return message;
            }
          
            /**                                                                                                                                                          
             * Create a new instance of {@link ConfigurationChangeException}.                                                                                            
             */
            public ConfigurationChangeException(String property,
                                                String newVal, String oldVal) {
              super(constructMessage(property, newVal, oldVal));
              this.property = property;
              this.newVal = newVal;
              this.oldVal = oldVal;
            }
          
            /**                                                                                                                                                          
             * Create a new instance of {@link ConfigurationChangeException}.                                                                                            
             */
            public ConfigurationChangeException(String property,
                                                String newVal, String oldVal,
                                                Throwable cause) {
              super(constructMessage(property, newVal, oldVal), cause);
              this.property = property;
              this.newVal = newVal;
              this.oldVal = oldVal;
            }
          
            /**                                                                                                                                                          
             * Get property that cannot be changed.                                                                                                                      
             */
            public String getProperty() {
              return property;
            }
          
            /**                                                                                                                                                          
             * Get value to which property was supposed to be changed.                                                                                                   
             */
            public String getNewValue() {
              return newVal;
            }
          
            /**                                                                                                                                                          
             * Get old value of property that cannot be changed.                                                                                                         
             */
            public String getOldValue() {
              return oldVal;
            }
          
          Show
          Patrick Kling added a comment - I absolutely agree that there will likely be many configuration options that cannot be changed at run time and implementing Reconfigurable does not require that we can make arbitrary changes. For the configuration changes that are possible at run time (such as the existing examples mentioned by Philip), it would be nice to have a consistent mechanism. The interface could also make it easier to propagate configuration changes (by calling changeConf on objects to which the current configuration was propagated). This should be a bit easier to read than the JavaDoc I posted before: package org.apache.hadoop.conf; /** * Something whose {@link Configuration} can be changed at run time. */ public interface Reconfigurable extends Configurable { /** * Change the configuration of this object. * * Change the configuration of this object to the configuration * passed as conf. * If it is not possible to change a configuration option, * a {@link ConfigurationChangeException} is thrown * and no changes are made to the current configuration */ void changeConf(Configuration conf) throws ConfigurationChangeException; } package org.apache.hadoop.conf; /** * Exception indicating that configuration property cannot be changed * at run time. */ public class ConfigurationChangeException extends Exception { private String property; private String newVal; private String oldVal; /** * Construct the exception message. */ private static String constructMessage( String property, String newVal, String oldVal) { String message = "Could not change property " + property; if (oldVal != null ) { message += " from " + oldVal; } if (newVal != null ) { message += " to " + newVal; } return message; } /** * Create a new instance of {@link ConfigurationChangeException}. */ public ConfigurationChangeException( String property, String newVal, String oldVal) { super (constructMessage(property, newVal, oldVal)); this .property = property; this .newVal = newVal; this .oldVal = oldVal; } /** * Create a new instance of {@link ConfigurationChangeException}. */ public ConfigurationChangeException( String property, String newVal, String oldVal, Throwable cause) { super (constructMessage(property, newVal, oldVal), cause); this .property = property; this .newVal = newVal; this .oldVal = oldVal; } /** * Get property that cannot be changed. */ public String getProperty() { return property; } /** * Get value to which property was supposed to be changed. */ public String getNewValue() { return newVal; } /** * Get old value of property that cannot be changed. */ public String getOldValue() { return oldVal; }
          Hide
          Philip Zeyliger added a comment -

          I think that the number of reconfigurable parameters will be quite small, and that they will require extra code. (e.g., changing the size of a thread pool requires a line or two of code). So maybe it's just better to expose those things in some more direct way? At the very least, we should make it quite clear where it's going to be documented what things are re-configurable.

          Show
          Philip Zeyliger added a comment - I think that the number of reconfigurable parameters will be quite small, and that they will require extra code. (e.g., changing the size of a thread pool requires a line or two of code). So maybe it's just better to expose those things in some more direct way? At the very least, we should make it quite clear where it's going to be documented what things are re-configurable.
          Hide
          Patrick Kling added a comment -

          I have uploaded a patch with the changes proposed in this JIRA. The included base class can be used to easily implement the interface by overriding the method changeProperty, which modifies a single configuration property:

          /**
           * Change a property or throw an exception.
           *
           * Subclasses should overrride this.
           * If newVal is null, change the property to its default value.
           */
          protected boolean changeProperty(String property, String newVal) 
            throws ConfigurationChangeException {
            throw new ConfigurationChangeException(property, newVal,
                                                   getConf().get(property));
            
          }
          

          This should enable us to include whatever code is necessary to make a configuration change. The overridden version of this method should also serve as a good documentation of what changes are supported by a particular class.

          Below is a version of the Reconfigurable interface with more detailed comments. Unfortunately, I cannot edit my previous comments, so I apologize if this is somewhat redundant:

          package org.apache.hadoop.conf;
          
          /**
           * Something whose {@link Configuration} can be changed at run time.
           */
          public interface Reconfigurable extends Configurable {
          
            /**
             * Change the configuration of this object.
             *
             * Change the configuration of this object  to the configuration
             * passed as conf.
             * If it is not possible to change a configuration option,
             * a {@link ConfigurationChangeException} is thrown
             * and no changes are made to the current configuration.
             *
             * Detailed semantics:
             * * If a property is set in the current configuration and in newConf, then 
             *   the configuration must be updated to reflect the value in newConf.
             * * If a property is not set in the current configuration but is set in
             *   newConf, then the configuration must be set to the value in newConf.
             * * If a property is set in the current configuration but is not set in 
             *   newConf, then the configuration must revert to the default value for
             *   this property.
             * * If the change required by these rules is not possible, a
             *   ConfigurationChangeException must be thrown. 
             */
            void changeConf(Configuration newConf) throws ConfigurationChangeException;
          }
          
          Show
          Patrick Kling added a comment - I have uploaded a patch with the changes proposed in this JIRA. The included base class can be used to easily implement the interface by overriding the method changeProperty, which modifies a single configuration property: /** * Change a property or throw an exception. * * Subclasses should overrride this . * If newVal is null , change the property to its default value. */ protected boolean changeProperty( String property, String newVal) throws ConfigurationChangeException { throw new ConfigurationChangeException(property, newVal, getConf().get(property)); } This should enable us to include whatever code is necessary to make a configuration change. The overridden version of this method should also serve as a good documentation of what changes are supported by a particular class. Below is a version of the Reconfigurable interface with more detailed comments. Unfortunately, I cannot edit my previous comments, so I apologize if this is somewhat redundant: package org.apache.hadoop.conf; /** * Something whose {@link Configuration} can be changed at run time. */ public interface Reconfigurable extends Configurable { /** * Change the configuration of this object. * * Change the configuration of this object to the configuration * passed as conf. * If it is not possible to change a configuration option, * a {@link ConfigurationChangeException} is thrown * and no changes are made to the current configuration. * * Detailed semantics: * * If a property is set in the current configuration and in newConf, then * the configuration must be updated to reflect the value in newConf. * * If a property is not set in the current configuration but is set in * newConf, then the configuration must be set to the value in newConf. * * If a property is set in the current configuration but is not set in * newConf, then the configuration must revert to the default value for * this property. * * If the change required by these rules is not possible, a * ConfigurationChangeException must be thrown. */ void changeConf(Configuration newConf) throws ConfigurationChangeException; }
          Hide
          Patrick Kling added a comment -

          Here is a new version of my patch. I have changed the interface Reconfigurable somewhat: Since only a subset of configuration properties can be changed at run time, it makes more sense to have an interface that allows us to change a single property at a time, rather than passing a Configuration object and then detecting all changes that need to be made. I have also added methods for detecting which configuration properties can be changed at runtime.

          /**                                                                                                                                                         
           * Something whose {@link Configuration} can be changed at run time.                                                                                        
           */
          public interface Reconfigurable extends Configurable {
          
            /**                                                                                                                                                       
             * Change a the configuration property on this object to the value specified.                                                                             
             *                                                                                                                                                        
             * Change a the configuration property on this object to the value specified                                                                              
             * and return the previous value that the configuration property was set to                                                                               
             * (or null if it was not previously set).                                                                                                                
             *                                                                                                                                                        
             * If the property cannot be changed, throw a                                                                                                             
             * {@link ConfigurationChangeException}.                                                                                                                  
             */
            public String changeConfProperty(String property, String newVal)
              throws ConfigurationChangeException;
          
            /**                                                                                                                                                       
             * Return whether a given property is changeable at run time.                                                                                             
             *                                                                                                                                                        
             * If isConfPropertyChangeable returns true for a property,                                                                                               
             * then changeConf should not throw an exception when changing                                                                                            
             * this property.                                                                                                                                         
             */
            public boolean isConfPropertyChangeable(String property);
          
            /**                                                                                                                                                       
             * Return all the properties that can be changed at run time.                                                                                             
             */
            public List<String> getChangeableConfProperties();
          }
          

          The patch also includes a simple servlet for changing the configuration of a Reconfigurable. In order to allow configuration changes at run time, I made all the getters and setters in Configuration synchronized.

          Show
          Patrick Kling added a comment - Here is a new version of my patch. I have changed the interface Reconfigurable somewhat: Since only a subset of configuration properties can be changed at run time, it makes more sense to have an interface that allows us to change a single property at a time, rather than passing a Configuration object and then detecting all changes that need to be made. I have also added methods for detecting which configuration properties can be changed at runtime. /** * Something whose {@link Configuration} can be changed at run time. */ public interface Reconfigurable extends Configurable { /** * Change a the configuration property on this object to the value specified. * * Change a the configuration property on this object to the value specified * and return the previous value that the configuration property was set to * (or null if it was not previously set). * * If the property cannot be changed, throw a * {@link ConfigurationChangeException}. */ public String changeConfProperty( String property, String newVal) throws ConfigurationChangeException; /** * Return whether a given property is changeable at run time. * * If isConfPropertyChangeable returns true for a property, * then changeConf should not throw an exception when changing * this property. */ public boolean isConfPropertyChangeable( String property); /** * Return all the properties that can be changed at run time. */ public List< String > getChangeableConfProperties(); } The patch also includes a simple servlet for changing the configuration of a Reconfigurable. In order to allow configuration changes at run time, I made all the getters and setters in Configuration synchronized.
          Hide
          dhruba borthakur added a comment -

          I added Doug and Tom to the list of watchers. I am hoping that they provide some feedback to this JIRA.

          Show
          dhruba borthakur added a comment - I added Doug and Tom to the list of watchers. I am hoping that they provide some feedback to this JIRA.
          Hide
          Doug Cutting added a comment -

          It doesn't seem unreasonable to me for some limited changes might be permitted without restarting. Restarting a namenode is not a lightweight operation.

          If the interface is 'Reconfigurable' then the method might be named 'reconfigureProp(String,String)', no?

          I don't like adding new interfaces that aren't yet used. So it might be good to file issues for at least one the daemons that you expect to implement this and at least draft those patches before this is committed. If we can't agree on some concrete usages of this interface within Hadoop, then there's no point in adding it.

          The servlet should probably return an HTTP error code when reconfiguration fails.

          A command-line client might be useful. Perhaps this will be added to, e.g.. the existing HDFS admin commands. But each command-line tool that uses this will share client-side HTTP call, handling of error codes, etc. So perhaps that should be encapsulated in a shared utility.

          Show
          Doug Cutting added a comment - It doesn't seem unreasonable to me for some limited changes might be permitted without restarting. Restarting a namenode is not a lightweight operation. If the interface is 'Reconfigurable' then the method might be named 'reconfigureProp(String,String)', no? I don't like adding new interfaces that aren't yet used. So it might be good to file issues for at least one the daemons that you expect to implement this and at least draft those patches before this is committed. If we can't agree on some concrete usages of this interface within Hadoop, then there's no point in adding it. The servlet should probably return an HTTP error code when reconfiguration fails. A command-line client might be useful. Perhaps this will be added to, e.g.. the existing HDFS admin commands. But each command-line tool that uses this will share client-side HTTP call, handling of error codes, etc. So perhaps that should be encapsulated in a shared utility.
          Hide
          Patrick Kling added a comment -

          As suggested by Doug, I have renamed the methods in Reconfigurable. I have also renamed ConfigurationChangeException to ReconfigurationException and ConfigurationChangeServlet to ReconfigurationServlet to keep everything consistent.

          Please see HDFS-1477 for a patch that makes NameNode Reconfigurable and allows us to change 2 properties.

          The servlet now returns error code 500 if a change fails.

          A command line tool for changing configuration properties sounds like a good idea but I am not quite sure how to provide a generic version of such a tool. If the command line tool runs locally, very little functionality beyond calling the interface methods would be needed. Maybe it would be best to leave command line utilities to a later JIRA once it becomes more clear if there is a need for additional support.

          Show
          Patrick Kling added a comment - As suggested by Doug, I have renamed the methods in Reconfigurable. I have also renamed ConfigurationChangeException to ReconfigurationException and ConfigurationChangeServlet to ReconfigurationServlet to keep everything consistent. Please see HDFS-1477 for a patch that makes NameNode Reconfigurable and allows us to change 2 properties. The servlet now returns error code 500 if a change fails. A command line tool for changing configuration properties sounds like a good idea but I am not quite sure how to provide a generic version of such a tool. If the command line tool runs locally, very little functionality beyond calling the interface methods would be needed. Maybe it would be best to leave command line utilities to a later JIRA once it becomes more clear if there is a need for additional support.
          Hide
          Todd Lipcon added a comment -

          I'm not sold on the interface being one of "set conf var 'foo' to value 'bar'". Internally, this seems fine, but exposing this as a servlet seems like it will encourage the case where runtime configuration diverges from what's in the XML configuration files on disk.

          Could we instead have the servlet/API be more similar to the existing refresh* functions, where it reloads the configuration off disk, and applies a diff? I'm imagining the servlet would show you a "preview" of the diff to be applied, along with a list of warnings for configuration variables that have been changed on disk but are not reconfigurable. Then the administrator can hit an "apply" button to reconfigure the settings that are reconfigurable.

          Show
          Todd Lipcon added a comment - I'm not sold on the interface being one of "set conf var 'foo' to value 'bar'". Internally, this seems fine, but exposing this as a servlet seems like it will encourage the case where runtime configuration diverges from what's in the XML configuration files on disk. Could we instead have the servlet/API be more similar to the existing refresh* functions, where it reloads the configuration off disk, and applies a diff? I'm imagining the servlet would show you a "preview" of the diff to be applied, along with a list of warnings for configuration variables that have been changed on disk but are not reconfigurable. Then the administrator can hit an "apply" button to reconfigure the settings that are reconfigurable.
          Hide
          Alejandro Abdelnur added a comment -

          I'd also prefer a refresh request that forces a reload of the config from disk.

          To detect diffs a memory copy of the current config could be used.

          Properties in the configuration XML should be flagged as reconfigurable (similar to final), if a property is not flagged as reconfigurable hot changes for that property are ignored and are not communicated via the Property listener (have you considered using the PropertyChangeListener API from the JDK?).

          Show
          Alejandro Abdelnur added a comment - I'd also prefer a refresh request that forces a reload of the config from disk. To detect diffs a memory copy of the current config could be used. Properties in the configuration XML should be flagged as reconfigurable (similar to final ), if a property is not flagged as reconfigurable hot changes for that property are ignored and are not communicated via the Property listener (have you considered using the PropertyChangeListener API from the JDK?).
          Hide
          dhruba borthakur added a comment -

          I too like the idea of having "bin/hadoop dfs -refreshConfig" that will reload the config from the conf file, diff the newly read ones with the ones in the NameNode.
          If the only ones that are different are Recofinfigurable, then the refreshConfig command will be successful, otherwise it is an error. The servlet could take a similar approach.

          Show
          dhruba borthakur added a comment - I too like the idea of having "bin/hadoop dfs -refreshConfig" that will reload the config from the conf file, diff the newly read ones with the ones in the NameNode. If the only ones that are different are Recofinfigurable, then the refreshConfig command will be successful, otherwise it is an error. The servlet could take a similar approach.
          Hide
          Patrick Kling added a comment -

          Here is an updated patch. Navigating to the ReconfigurationServlet for a Reconfigurable node now reloads the configuration files from disk and shows a list of all properties that have changed (along with their old and new values). If there are any changes to properties that cannot be reconfigured at run time, a warning is shown. After reviewing the changes, the admin can approve them, at which point they are applied through the methods in the Reconfigurable interface.

          The code for computing the diff between the old and the new configuration is in ReconfigurationUtil, which should make it easy to add this functionality to existing command line tools.

          For this revision, I didn't end up going with the PropertyChangeListener API, since there doesn't seem to be an ideal source object with which to register listeners while still keeping this change generic enough to be useful for all configured nodes in the system.

          The review board revision corresponding to this patch can be found at: https://reviews.apache.org/r/21/

          Show
          Patrick Kling added a comment - Here is an updated patch. Navigating to the ReconfigurationServlet for a Reconfigurable node now reloads the configuration files from disk and shows a list of all properties that have changed (along with their old and new values). If there are any changes to properties that cannot be reconfigured at run time, a warning is shown. After reviewing the changes, the admin can approve them, at which point they are applied through the methods in the Reconfigurable interface. The code for computing the diff between the old and the new configuration is in ReconfigurationUtil, which should make it easy to add this functionality to existing command line tools. For this revision, I didn't end up going with the PropertyChangeListener API, since there doesn't seem to be an ideal source object with which to register listeners while still keeping this change generic enough to be useful for all configured nodes in the system. The review board revision corresponding to this patch can be found at: https://reviews.apache.org/r/21/
          Hide
          Patrick Kling added a comment -

          Incorporated Todd's comments on the review board.

          • Added unit tests for ReconfigurationUtil.getChangedProperties, Reconfigurable interface methods and cross-thread visibility of configuration changes.
          • Added proper escaping to servlet.
          • Removed synchronization from Configuration methods that were already protected by synchronization of getProps(). In this revision, I have only added synchronization to the setter methods.
          • Changed getReconfigurableProperties to return Collection rather than List.
          • Fixed typo.
          • Removed commented-out code.
          • Removed unnecessary imports.
          Show
          Patrick Kling added a comment - Incorporated Todd's comments on the review board. Added unit tests for ReconfigurationUtil.getChangedProperties, Reconfigurable interface methods and cross-thread visibility of configuration changes. Added proper escaping to servlet. Removed synchronization from Configuration methods that were already protected by synchronization of getProps(). In this revision, I have only added synchronization to the setter methods. Changed getReconfigurableProperties to return Collection rather than List. Fixed typo. Removed commented-out code. Removed unnecessary imports.
          Hide
          Patrick Kling added a comment -

          Any further comments on this? I would appreciate if someone could review my revised patch at https://reviews.apache.org/r/21/ .

          Show
          Patrick Kling added a comment - Any further comments on this? I would appreciate if someone could review my revised patch at https://reviews.apache.org/r/21/ .
          Hide
          dhruba borthakur added a comment -

          Doug, Todd, Alejandro: any new comments/feedback on this JIRA? Thanks.

          Show
          dhruba borthakur added a comment - Doug, Todd, Alejandro: any new comments/feedback on this JIRA? Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12458671/HADOOP-7001.4.patch
          against trunk revision 1036130.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          -1 javac. The applied patch generated 1051 javac compiler warnings (more than the trunk's current 1048 warnings).

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/109//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/109//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/109//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12458671/HADOOP-7001.4.patch against trunk revision 1036130. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 1051 javac compiler warnings (more than the trunk's current 1048 warnings). -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/109//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/109//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/109//console This message is automatically generated.
          Hide
          Patrick Kling added a comment -

          Fixed warnings.

          Show
          Patrick Kling added a comment - Fixed warnings.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12459978/HADOOP-7001.5.patch
          against trunk revision 1036746.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/116//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/116//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/116//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459978/HADOOP-7001.5.patch against trunk revision 1036746. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/116//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/116//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/116//console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          +1, code looks good to me. I will commit it.

          Show
          dhruba borthakur added a comment - +1, code looks good to me. I will commit it.
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Patrick!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Patrick!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #445 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/445/)
          HADOOP-7001. Configuration changes can occur via the Reconfigurable interface. (Patrick Kline via dhruba and nigel)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #445 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/445/ ) HADOOP-7001 . Configuration changes can occur via the Reconfigurable interface. (Patrick Kline via dhruba and nigel)
          Hide
          Konstantin Boudnik added a comment -

          I am sorry to jump in so late, but the test are done for a mix of JUnit v3 and v4. They should be for v4 only (e.g. inheritance of TestCase needs to be eliminated). It can be done via separate JIRA or as a second commit for this one (I personally favor latter less).

          Show
          Konstantin Boudnik added a comment - I am sorry to jump in so late, but the test are done for a mix of JUnit v3 and v4. They should be for v4 only (e.g. inheritance of TestCase needs to be eliminated). It can be done via separate JIRA or as a second commit for this one (I personally favor latter less).
          Hide
          Konstantin Boudnik added a comment -

          One more thing: asserts have to have meaningful messages to them. It makes failure analysis a way easier.

          Show
          Konstantin Boudnik added a comment - One more thing: asserts have to have meaningful messages to them. It makes failure analysis a way easier.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #524 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/524/)
          HADOOP-7001. Configuration changes can occur via the Reconfigurable interface. (Patrick Kline via dhruba and nigel)
          HADOOP-7001. Configuration changes can occur via the Reconfigurable
          interface. (Patrick Kline via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #524 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/524/ ) HADOOP-7001 . Configuration changes can occur via the Reconfigurable interface. (Patrick Kline via dhruba and nigel) HADOOP-7001 . Configuration changes can occur via the Reconfigurable interface. (Patrick Kline via dhruba)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #60 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/60/)
          Merging changes r1038493 and r1038480 for HADOOP-7001 from trunk

          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130306
          Files :

          • /hadoop/common/branches/branch-0.22/src/docs
          • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurableBase.java
          • /hadoop/common/branches/branch-0.22
          • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Reconfigurable.java
          • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationException.java
          • /hadoop/common/branches/branch-0.22/src/test/core
          • /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java
          • /hadoop/common/branches/branch-0.22/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/src/java
          • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/conf/TestReconfiguration.java
          • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationUtil.java
          • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationServlet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #60 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/60/ ) Merging changes r1038493 and r1038480 for HADOOP-7001 from trunk suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130306 Files : /hadoop/common/branches/branch-0.22/src/docs /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurableBase.java /hadoop/common/branches/branch-0.22 /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Reconfigurable.java /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationException.java /hadoop/common/branches/branch-0.22/src/test/core /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java /hadoop/common/branches/branch-0.22/CHANGES.txt /hadoop/common/branches/branch-0.22/src/java /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/conf/TestReconfiguration.java /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationUtil.java /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationServlet.java
          Hide
          su wenbo added a comment -

          Excuse me, I am a newer, I tried to use your patch in version 1.0.3. Dose it work? And how to make it work?

          Show
          su wenbo added a comment - Excuse me, I am a newer, I tried to use your patch in version 1.0.3. Dose it work? And how to make it work?
          Hide
          Harsh J added a comment -

          Hi Su,

          This patch, that merely lays the foundation for reconfigurable elements, is not applicable on the branch-1 releases. It is pre-present in the branch-2 release (2.0.0, 2.0.1, 2.0.2, etc.).

          Show
          Harsh J added a comment - Hi Su, This patch, that merely lays the foundation for reconfigurable elements, is not applicable on the branch-1 releases. It is pre-present in the branch-2 release (2.0.0, 2.0.1, 2.0.2, etc.).

            People

            • Assignee:
              Patrick Kling
              Reporter:
              Patrick Kling
            • Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development