[HADOOP-3582] Improve how Hadoop gets configured

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: conf
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 17/Jun/08 12:37
• Updated: 23/Mar/15 16:19

Description

Improve how Hadoop gets configured

The current approach of a two-level XML configuration file works; it offers

• default values in (easily overridden) configuration files, rather than just Java source
• A way to override the default values
• conversion from string to float, double.
• with the ${property} evaluation, there is some ability to reference system values for some limited adaptation.
• errors show up at parse time (except for value parse problems)
• A serialization format to exchange configuration with other nodes
• the possibility of updating a local (in-memory) configuration

But it has limits
[1] Requires a change to the XML files to be pushed out to every node
[2] Differences between configurations can cause obscure bugs
[3] No support for complex hierarchical configurations
[4] No easy way to cross-reference data other than copy and paste.
[5] No way for a deployed instance to update configuration data for other instances to query
[6] Value type checking/dereferencing failure is not signalled by a custom error; there is no explicit exception on any of the get/set operations.
[7] No consistency with names.
[8] Not easily managed by different configuration architectures/tools

This issue is to group/track the problems, then discuss solutions.

Issue Links

• incorporates: HADOOP-2366 Space in the value for dfs.data.dir can cause great problems
• incorporates: HADOOP-3583 Pull out configuration element names from inline code to interface classes for use across Hadoop and other apps
• incorporates: HADOOP-3579 Support property groups in Hadoop configuration
• incorporates: HADOOP-3584 Add an explicit HadoopConfigurationException that extends RuntimeException
• incorporates: HADOOP-24 make Configuration an interface
• relates to: HADOOP-3287 Being able to set default job configuration values on the jobtracker
• relates to: HADOOP-5670 Hadoop configurations should be read from a distributed system
• relates to: HDFS-310 Validate configuration parameters
• relates to: MAPREDUCE-475 JobConf should validate key names in well-defined namespaces and warn on misspelling
• relates to: HADOOP-1307 Classification of various configuration knobs