[HADOOP-8031] Configuration class fails to find embedded .jar resources; should use URL.openStream()

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 2.0.2-alpha
• Component/s: conf
• Hadoop Flags: Reviewed
• Assignee: Elias Ross
• Reporter: Elias Ross
• Created: 07/Feb/12 20:22
• Updated: 11/Oct/12 17:45
• Resolved: 30/Aug/12 20:16

Description

While running a hadoop client within RHQ (monitoring software) using its classloader, I see this:

2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2] (org.apache.hadoop.conf.Configuration)- parsing jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failed to start component for Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start component for resource Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)

This is because the URL

jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml

cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs are for an old version of Configuration class, but the new version has the same code.)

The solution is to obtain the resource stream directly from the URL object itself.

That is to say:

         URL url = getResource((String)name);
-        if (url != null) {
-          if (!quiet) {
-            LOG.info("parsing " + url);
-          }
-          doc = builder.parse(url.toString());
-        }
+        doc = builder.parse(url.openStream());

Note: I have a full patch pending approval at Apple for this change, including some cleanup.

Issue Links

• is related to: HADOOP-8749 HADOOP-8031 changed the way in which relative xincludes are handled in Configuration.