Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
class JobConf (or its base class Configuration)
should be extended to enable enumeration of all its key-value pairs.
( more precisely: the Properties returned by Configuration.getProps() )
This will be useful to "export" all JobConf properties to environment variables.
We use env.vars to expose some Hadoop context to non-Java MapReduce applications.
Note that the typed properties are also represented as Strings
(getInt, getStrings, getClass, etc.)
So a single enumeration exposes everything as (untyped) environment variables.
The proposed escaping rules from JobConf properties to env.var are:
1. values are left as-is.
2. keys are escaped as follows:
[A-Za-z0-9] --> unchanged.
all other chars --> underscore.
For example
set("mapred.input.key.class", "com.example.MyKey")
becomes env.var:
export mapred_input_key_class=com.example.MyKey
Justification:
1. Environment variables are case-sensitive. (Although uppercase is the preferred convention)
So no need to uppercase everything.
2. Some characters are forbidden in env.vars, or at least not shell-friendly:
For example period, colon are problematic.
3. The Hadoop conventions are already hierarchical and provide some namespace protection.
This means we don't need an additional prefix as protection.
For example all exported environment variables will start with "mapred." , "dfs." , "ipc." etc.
This means they will not conflict with standard environemnt variables like PATH, USER, etc.
And they will not conflict with standard hadoop env.vars because those are upper-case. (like HADOOP_CONF_DIR)