Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8031

Configuration class fails to find embedded .jar resources; should use URL.openStream()

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.0.2-alpha
    • conf
    • None
    • Reviewed

    Description

      While running a hadoop client within RHQ (monitoring software) using its classloader, I see this:

      2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2] (org.apache.hadoop.conf.Configuration)- parsing jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
      2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failed to start component for Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
      org.rhq.core.clientapi.agent.PluginContainerException: Failed to start component for resource Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com].
      Caused by: java.lang.RuntimeException: core-site.xml not found
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)

      This is because the URL

      jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml

      cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs are for an old version of Configuration class, but the new version has the same code.)

      The solution is to obtain the resource stream directly from the URL object itself.

      That is to say:

               URL url = getResource((String)name);
      -        if (url != null) {
      -          if (!quiet) {
      -            LOG.info("parsing " + url);
      -          }
      -          doc = builder.parse(url.toString());
      -        }
      +        doc = builder.parse(url.openStream());
      

      Note: I have a full patch pending approval at Apple for this change, including some cleanup.

      Attachments

        1. 0001-fix-HADOOP-7982-class-loader.patch
          3 kB
          Elias Ross
        2. hadoop-8031.txt
          3 kB
          Eli Collins
        3. HADOOP-8031.patch
          3 kB
          Ahmed Radwan
        4. HADOOP-8031-part2.patch
          2 kB
          Ahmed Radwan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            genman Elias Ross
            genman Elias Ross
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment