Hadoop Common
  1. Hadoop Common
  2. HADOOP-8031

Configuration class fails to find embedded .jar resources; should use URL.openStream()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: conf
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      While running a hadoop client within RHQ (monitoring software) using its classloader, I see this:

      2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2] (org.apache.hadoop.conf.Configuration)- parsing jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
      2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failed to start component for Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
      org.rhq.core.clientapi.agent.PluginContainerException: Failed to start component for resource Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com].
      Caused by: java.lang.RuntimeException: core-site.xml not found
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)

      This is because the URL

      jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml

      cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs are for an old version of Configuration class, but the new version has the same code.)

      The solution is to obtain the resource stream directly from the URL object itself.

      That is to say:

               URL url = getResource((String)name);
      -        if (url != null) {
      -          if (!quiet) {
      -            LOG.info("parsing " + url);
      -          }
      -          doc = builder.parse(url.toString());
      -        }
      +        doc = builder.parse(url.openStream());
      

      Note: I have a full patch pending approval at Apple for this change, including some cleanup.

      1. 0001-fix-HADOOP-7982-class-loader.patch
        3 kB
        Elias Ross
      2. HADOOP-8031.patch
        3 kB
        Ahmed Radwan
      3. hadoop-8031.txt
        3 kB
        Eli Collins
      4. HADOOP-8031-part2.patch
        2 kB
        Ahmed Radwan

        Issue Links

          Activity

          Elias Ross created issue -
          Elias Ross made changes -
          Field Original Value New Value
          Attachment 0001-fix-HADOOP-7982-class-loader.patch [ 12517303 ]
          Eli Collins made changes -
          Assignee Elias Ross [ genman ]
          Eli Collins made changes -
          Attachment hadoop-8031.txt [ 12529840 ]
          Eli Collins made changes -
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Target Version/s 2.0.1-alpha [ 12321441 ]
          Component/s conf [ 12310711 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-8031.patch [ 12542098 ]
          Ahmed Radwan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.2.0-alpha [ 12322473 ]
          Resolution Fixed [ 1 ]
          Todd Lipcon made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Ahmed Radwan made changes -
          Attachment HADOOP-8031-part2.patch [ 12543031 ]
          Todd Lipcon made changes -
          Link This issue is related to HADOOP-8749 [ HADOOP-8749 ]
          Todd Lipcon made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Elias Ross
              Reporter:
              Elias Ross
            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development