Details
-
Improvement
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
2.7.1
-
None
-
None
Description
Problem
It is currently not possible to add default resources to org.apache.hadoop.conf.Configuration that are not included in the class path.
We need to add resources to org.apache.hadoop.conf.Configuration that are not included in the class path. Currently, the only way to do this is to use addResource(URL) or addResource(Path) which both are instance methods. However, a library we work with creates Configuration instances internally that we cannot access. So we need to add them as default resources in order to make them available to all subsequent Configuration instances (and those existing instances where loadDefaults equals true, of course).
Solution
The proposed solution is to add the following static methods:
public static synchronized void addDefaultResource(URL url); public static synchronized void addDefaultResource(Path file);
Please see the attached patch for details.
Example
In order to initialize the Configuration correctly, what we need to do currently is as follows:
// Set resource paths for the class loader File ffile = new File("/path/to/conf/dir"); URL url = ffile.toURI().toURL(); URLClassLoader systemClassLoader = (URLClassLoader) getClass().getClassLoader(); Class<URLClassLoader> classLoaderClass = URLClassLoader.class; Method method = classLoaderClass.getDeclaredMethod("addURL", new Class[]{URL.class}); method.setAccessible(true); method.invoke(systemClassLoader, new Object[]{url}); //----------------------------- // Read cluster config for (File f : ffile.listFiles()) { if(f.getName().toLowerCase().endsWith(".xml")) { Configuration.addDefaultResource(f.getName()); } }
As you can see, we currently need to manipulate the class path through reflection which is kind of dirty.
With the proposed solution, the same (adding default resources from outside the class path) can be achieved doing the following:
// Set resource paths for the class loader File ffile = new File("/path/to/conf/dir"); //----------------------------- // Read cluster config for (File f : ffile.listFiles()) { if(f.getName().toLowerCase().endsWith(".xml")) { Configuration.addDefaultResource(new Path(f.getAbsolutePath())); } }