Hadoop Common
  1. Hadoop Common
  2. HADOOP-5254

Xinclude setup results in a stack trace

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0, 0.21.0
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None
    • Environment:

      java version "1.6.0_02"
      Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
      BEA JRockit(R) (build R27.4.0-90-89592-1.6.0_02-20070928-1715-linux-x86_64, compiled mode)

    • Hadoop Flags:
      Reviewed

      Description

      seen this in SVN_HEAD, and it was mentioned on the user list in the week. It explains why my health tests are failing on class.ForName(FSConstants)

      gistration(127.0.0.1:8024, storageID=DS-1466307248-127.0.1.1-8024-1234537374021, infoPort=8022, ipcPort=50020):DataXceiver
      [sf-startdaemon-debug] java.lang.ExceptionInInitializerError
      [sf-startdaemon-debug] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:76)
      [sf-startdaemon-debug] at java.lang.Thread.run(Thread.java:619)
      [sf-startdaemon-debug] Caused by: java.lang.UnsupportedOperationException: This parser does not support specification "null" version "null"
      [sf-startdaemon-debug] at javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)

        Issue Links

          Activity

          Hide
          Shevek added a comment -

          I hit this issue this week, and having waded through 350Mb of logfiles and this amongst other tickets, I wrote this library to help diagnose this class of issue:

          https://github.com/shevek/jdiagnostics

          The 350Mb is because Hadoop logs the stack trace to the ParserConfigurationException every time it passes that code path, which is sufficiently painful and impactful on performance that I've modified my local copy to fast-fail, and then I diagnose it.

          Hope it's useful.

          The ant -diagnostics thing is a red herring, because while ant looks in /usr/share/java, Hadoop doesn't.

          Show
          Shevek added a comment - I hit this issue this week, and having waded through 350Mb of logfiles and this amongst other tickets, I wrote this library to help diagnose this class of issue: https://github.com/shevek/jdiagnostics The 350Mb is because Hadoop logs the stack trace to the ParserConfigurationException every time it passes that code path, which is sufficiently painful and impactful on performance that I've modified my local copy to fast-fail, and then I diagnose it. Hope it's useful. The ant -diagnostics thing is a red herring, because while ant looks in /usr/share/java, Hadoop doesn't.
          Hide
          Steve Loughran added a comment -

          The XML parser printout implies that you are running on some linux distribution that has stuck a copy of xerces into /user/share/java.

          XML Parser : org.apache.xerces.jaxp.SAXParserImpl
          XML Parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar
          Namespace-aware parser : org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser
          Namespace-aware parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar

          Try renaming that file and

          1. seeing if that makes the problem goes away
          2. what shows up in ant -diagnostics for XML parsers once it has been renamed

          Given that Xerces 2.9.1 is XML-include-aware, I'm not sure where the real problem lies here. There is probably another XML parser getting into your application's class path. Have a look at http://xml.apache.org/xalan-j/usagepatterns.html to see if it offers any insight.

          What you can do here is have an entry point, a Main method, that tries to create an XML parser and not only prints the classname it uses, it gets the URL of it (via .getClass().getClassLoader().getResource("org/apache/xerces/jaxp/SAXParserImpl").getURL().toExternalForm() ); (replacing the resource path with the name of whichever XML parser you get.

          Ant's <whichresource> task can do this for you -just hand it the classpath of your app and the classname you want to locate

          Show
          Steve Loughran added a comment - The XML parser printout implies that you are running on some linux distribution that has stuck a copy of xerces into /user/share/java. XML Parser : org.apache.xerces.jaxp.SAXParserImpl XML Parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar Namespace-aware parser : org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser Namespace-aware parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar Try renaming that file and seeing if that makes the problem goes away what shows up in ant -diagnostics for XML parsers once it has been renamed Given that Xerces 2.9.1 is XML-include-aware, I'm not sure where the real problem lies here. There is probably another XML parser getting into your application's class path. Have a look at http://xml.apache.org/xalan-j/usagepatterns.html to see if it offers any insight. What you can do here is have an entry point, a Main method, that tries to create an XML parser and not only prints the classname it uses, it gets the URL of it (via .getClass().getClassLoader().getResource("org/apache/xerces/jaxp/SAXParserImpl").getURL().toExternalForm() ); (replacing the resource path with the name of whichever XML parser you get. Ant's <whichresource> task can do this for you -just hand it the classpath of your app and the classname you want to locate
          Hide
          fodil added a comment -

          Here is the result of my ant -diagnostics

          Unable to locate tools.jar. Expected to find it in /usr/lib/jvm/java-6-sun-1.6.0.26/lib/tools.jar
          ------- Ant diagnostics report -------
          Apache Ant version 1.8.0 compiled on April 9 2010

          -------------------------------------------
          Implementation Version
          -------------------------------------------
          core tasks : 1.8.0 in file:/usr/share/ant/lib/ant.jar
          optional tasks : 1.8.0 in file:/usr/share/ant/lib/ant-nodeps.jar

          -------------------------------------------
          ANT PROPERTIES
          -------------------------------------------
          ant.version: Apache Ant version 1.8.0 compiled on April 9 2010
          ant.java.version: 1.6
          ant.core.lib: /usr/share/ant/lib/ant.jar
          ant.home: /usr/share/ant

          -------------------------------------------
          ANT_HOME/lib jar listing
          -------------------------------------------
          ant.home: /usr/share/ant
          ant-stylebook.jar (2332 bytes)
          ant-jsch.jar (40162 bytes)
          ant-apache-resolver.jar (4070 bytes)
          ant-commons-net.jar (85060 bytes)
          ant-junit.jar (101315 bytes)
          ant-jmf.jar (6723 bytes)
          ant-commons-logging.jar (3910 bytes)
          ant-apache-xalan2.jar (2292 bytes)
          ant-antlr.jar (5753 bytes)
          ant-apache-bcel.jar (8764 bytes)
          ant-apache-bsf.jar (3940 bytes)
          ant-apache-regexp.jar (3762 bytes)
          ant-nodeps.jar (402286 bytes)
          ant-apache-log4j.jar (3045 bytes)
          ant-apache-oro.jar (39622 bytes)
          ant-jdepend.jar (8276 bytes)
          ant-launcher.jar (12276 bytes)
          ant-trax.jar (8391 bytes)
          ant-javamail.jar (7856 bytes)
          ant-swing.jar (7542 bytes)
          ant.jar (1508803 bytes)

          -------------------------------------------
          USER_HOME/.ant/lib jar listing
          -------------------------------------------
          user.home: /home/hadoop
          No such directory.

          -------------------------------------------
          Tasks availability
          -------------------------------------------
          image : Not Available (the implementation class is not present)
          sshexec : Missing dependency com.jcraft.jsch.Logger
          wlrun : Not Available (the implementation class is not present)
          scp : Missing dependency com.jcraft.jsch.Logger
          stlist : Not Available (the implementation class is not present)
          sshsession : Missing dependency com.jcraft.jsch.Logger
          netrexxc : Not Available (the implementation class is not present)
          starteam : Not Available (the implementation class is not present)
          stlabel : Not Available (the implementation class is not present)
          jdepend : Missing dependency jdepend.xmlui.JDepend
          stcheckin : Not Available (the implementation class is not present)
          stcheckout : Not Available (the implementation class is not present)
          ejbc : Not Available (the implementation class is not present)
          gjdoc : Not Available (the implementation class is not present)
          wlstop : Not Available (the implementation class is not present)
          ddcreator : Not Available (the implementation class is not present)
          A task being missing/unavailable should only matter if you are trying to use it

          -------------------------------------------
          org.apache.env.Which diagnostics
          -------------------------------------------
          Not available.
          Download it at http://xml.apache.org/commons/

          -------------------------------------------
          XML Parser information
          -------------------------------------------
          XML Parser : org.apache.xerces.jaxp.SAXParserImpl
          XML Parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar
          Namespace-aware parser : org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser
          Namespace-aware parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar

          -------------------------------------------
          XSLT Processor information
          -------------------------------------------
          XSLT Processor : com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl
          XSLT Processor Location: unknown

          -------------------------------------------
          System properties
          -------------------------------------------
          java.runtime.name : Java(TM) SE Runtime Environment
          sun.boot.library.path : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64
          java.vm.version : 20.1-b02
          ant.library.dir : /usr/share/ant/lib
          java.vm.vendor : Sun Microsystems Inc.
          java.vendor.url : http://java.sun.com/
          path.separator : :
          java.vm.name : Java HotSpot(TM) 64-Bit Server VM
          file.encoding.pkg : sun.io
          user.country : US
          sun.java.launcher : SUN_STANDARD
          sun.os.patch.level : unknown
          java.vm.specification.name : Java Virtual Machine Specification
          user.dir : /home/hadoop
          java.runtime.version : 1.6.0_26-b03
          java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
          java.endorsed.dirs : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/endorsed
          os.arch : amd64
          java.io.tmpdir : /tmp
          line.separator :

          java.vm.specification.vendor : Sun Microsystems Inc.
          os.name : Linux
          ant.home : /usr/share/ant
          sun.jnu.encoding : UTF-8
          java.library.path : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
          java.specification.name : Java Platform API Specification
          java.class.version : 50.0
          sun.management.compiler : HotSpot 64-Bit Tiered Compilers
          os.version : 2.6.32-33-generic
          user.home : /home/hadoop
          user.timezone : America/New_York
          java.awt.printerjob : sun.print.PSPrinterJob
          file.encoding : UTF-8
          java.specification.version : 1.6
          user.name : hadoop
          java.class.path : /usr/share/ant/lib/ant-launcher.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/xercesImpl.jar:/usr/share/ant/lib/ant-stylebook.jar:/usr/share/ant/lib/ant-jsch.jar:/usr/share/ant/lib/ant-apache-resolver.jar:/usr/share/ant/lib/ant-commons-net.jar:/usr/share/ant/lib/ant-junit.jar:/usr/share/ant/lib/ant-jmf.jar:/usr/share/ant/lib/ant-commons-logging.jar:/usr/share/ant/lib/ant-apache-xalan2.jar:/usr/share/ant/lib/ant-antlr.jar:/usr/share/ant/lib/ant-apache-bcel.jar:/usr/share/ant/lib/ant-apache-bsf.jar:/usr/share/ant/lib/ant-apache-regexp.jar:/usr/share/ant/lib/ant-nodeps.jar:/usr/share/ant/lib/ant-apache-log4j.jar:/usr/share/ant/lib/ant-apache-oro.jar:/usr/share/ant/lib/ant-jdepend.jar:/usr/share/ant/lib/ant-launcher.jar:/usr/share/ant/lib/ant-trax.jar:/usr/share/ant/lib/ant-javamail.jar:/usr/share/ant/lib/ant-swing.jar:/usr/share/ant/lib/ant.jar
          java.vm.specification.version : 1.0
          sun.arch.data.model : 64
          java.home : /usr/lib/jvm/java-6-sun-1.6.0.26/jre
          sun.java.command : org.apache.tools.ant.launch.Launcher -cp -diagnostics
          java.specification.vendor : Sun Microsystems Inc.
          user.language : en
          java.vm.info : mixed mode
          java.version : 1.6.0_26
          java.ext.dirs : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ext:/usr/java/packages/lib/ext
          sun.boot.class.path : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/charsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes
          java.vendor : Sun Microsystems Inc.
          file.separator : /
          java.vendor.url.bug : http://java.sun.com/cgi-bin/bugreport.cgi
          sun.cpu.endian : little
          sun.io.unicode.encoding : UnicodeLittle
          sun.desktop : gnome
          sun.cpu.isalist :

          -------------------------------------------
          Temp dir
          -------------------------------------------
          Temp dir is /tmp
          Temp dir is writeable
          Temp dir alignment with system clock is -555 ms

          -------------------------------------------
          Locale information
          -------------------------------------------
          Timezone Eastern Standard Time offset=-14400000

          -------------------------------------------
          Proxy information
          -------------------------------------------
          Java1.5+ proxy settings:
          Direct connection

          Show
          fodil added a comment - Here is the result of my ant -diagnostics Unable to locate tools.jar. Expected to find it in /usr/lib/jvm/java-6-sun-1.6.0.26/lib/tools.jar ------- Ant diagnostics report ------- Apache Ant version 1.8.0 compiled on April 9 2010 ------------------------------------------- Implementation Version ------------------------------------------- core tasks : 1.8.0 in file:/usr/share/ant/lib/ant.jar optional tasks : 1.8.0 in file:/usr/share/ant/lib/ant-nodeps.jar ------------------------------------------- ANT PROPERTIES ------------------------------------------- ant.version: Apache Ant version 1.8.0 compiled on April 9 2010 ant.java.version: 1.6 ant.core.lib: /usr/share/ant/lib/ant.jar ant.home: /usr/share/ant ------------------------------------------- ANT_HOME/lib jar listing ------------------------------------------- ant.home: /usr/share/ant ant-stylebook.jar (2332 bytes) ant-jsch.jar (40162 bytes) ant-apache-resolver.jar (4070 bytes) ant-commons-net.jar (85060 bytes) ant-junit.jar (101315 bytes) ant-jmf.jar (6723 bytes) ant-commons-logging.jar (3910 bytes) ant-apache-xalan2.jar (2292 bytes) ant-antlr.jar (5753 bytes) ant-apache-bcel.jar (8764 bytes) ant-apache-bsf.jar (3940 bytes) ant-apache-regexp.jar (3762 bytes) ant-nodeps.jar (402286 bytes) ant-apache-log4j.jar (3045 bytes) ant-apache-oro.jar (39622 bytes) ant-jdepend.jar (8276 bytes) ant-launcher.jar (12276 bytes) ant-trax.jar (8391 bytes) ant-javamail.jar (7856 bytes) ant-swing.jar (7542 bytes) ant.jar (1508803 bytes) ------------------------------------------- USER_HOME/.ant/lib jar listing ------------------------------------------- user.home: /home/hadoop No such directory. ------------------------------------------- Tasks availability ------------------------------------------- image : Not Available (the implementation class is not present) sshexec : Missing dependency com.jcraft.jsch.Logger wlrun : Not Available (the implementation class is not present) scp : Missing dependency com.jcraft.jsch.Logger stlist : Not Available (the implementation class is not present) sshsession : Missing dependency com.jcraft.jsch.Logger netrexxc : Not Available (the implementation class is not present) starteam : Not Available (the implementation class is not present) stlabel : Not Available (the implementation class is not present) jdepend : Missing dependency jdepend.xmlui.JDepend stcheckin : Not Available (the implementation class is not present) stcheckout : Not Available (the implementation class is not present) ejbc : Not Available (the implementation class is not present) gjdoc : Not Available (the implementation class is not present) wlstop : Not Available (the implementation class is not present) ddcreator : Not Available (the implementation class is not present) A task being missing/unavailable should only matter if you are trying to use it ------------------------------------------- org.apache.env.Which diagnostics ------------------------------------------- Not available. Download it at http://xml.apache.org/commons/ ------------------------------------------- XML Parser information ------------------------------------------- XML Parser : org.apache.xerces.jaxp.SAXParserImpl XML Parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar Namespace-aware parser : org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser Namespace-aware parser Location: file:/usr/share/java/xercesImpl-2.9.1.jar ------------------------------------------- XSLT Processor information ------------------------------------------- XSLT Processor : com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl XSLT Processor Location: unknown ------------------------------------------- System properties ------------------------------------------- java.runtime.name : Java(TM) SE Runtime Environment sun.boot.library.path : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64 java.vm.version : 20.1-b02 ant.library.dir : /usr/share/ant/lib java.vm.vendor : Sun Microsystems Inc. java.vendor.url : http://java.sun.com/ path.separator : : java.vm.name : Java HotSpot(TM) 64-Bit Server VM file.encoding.pkg : sun.io user.country : US sun.java.launcher : SUN_STANDARD sun.os.patch.level : unknown java.vm.specification.name : Java Virtual Machine Specification user.dir : /home/hadoop java.runtime.version : 1.6.0_26-b03 java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment java.endorsed.dirs : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/endorsed os.arch : amd64 java.io.tmpdir : /tmp line.separator : java.vm.specification.vendor : Sun Microsystems Inc. os.name : Linux ant.home : /usr/share/ant sun.jnu.encoding : UTF-8 java.library.path : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.specification.name : Java Platform API Specification java.class.version : 50.0 sun.management.compiler : HotSpot 64-Bit Tiered Compilers os.version : 2.6.32-33-generic user.home : /home/hadoop user.timezone : America/New_York java.awt.printerjob : sun.print.PSPrinterJob file.encoding : UTF-8 java.specification.version : 1.6 user.name : hadoop java.class.path : /usr/share/ant/lib/ant-launcher.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/xercesImpl.jar:/usr/share/ant/lib/ant-stylebook.jar:/usr/share/ant/lib/ant-jsch.jar:/usr/share/ant/lib/ant-apache-resolver.jar:/usr/share/ant/lib/ant-commons-net.jar:/usr/share/ant/lib/ant-junit.jar:/usr/share/ant/lib/ant-jmf.jar:/usr/share/ant/lib/ant-commons-logging.jar:/usr/share/ant/lib/ant-apache-xalan2.jar:/usr/share/ant/lib/ant-antlr.jar:/usr/share/ant/lib/ant-apache-bcel.jar:/usr/share/ant/lib/ant-apache-bsf.jar:/usr/share/ant/lib/ant-apache-regexp.jar:/usr/share/ant/lib/ant-nodeps.jar:/usr/share/ant/lib/ant-apache-log4j.jar:/usr/share/ant/lib/ant-apache-oro.jar:/usr/share/ant/lib/ant-jdepend.jar:/usr/share/ant/lib/ant-launcher.jar:/usr/share/ant/lib/ant-trax.jar:/usr/share/ant/lib/ant-javamail.jar:/usr/share/ant/lib/ant-swing.jar:/usr/share/ant/lib/ant.jar java.vm.specification.version : 1.0 sun.arch.data.model : 64 java.home : /usr/lib/jvm/java-6-sun-1.6.0.26/jre sun.java.command : org.apache.tools.ant.launch.Launcher -cp -diagnostics java.specification.vendor : Sun Microsystems Inc. user.language : en java.vm.info : mixed mode java.version : 1.6.0_26 java.ext.dirs : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ext:/usr/java/packages/lib/ext sun.boot.class.path : /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/charsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes java.vendor : Sun Microsystems Inc. file.separator : / java.vendor.url.bug : http://java.sun.com/cgi-bin/bugreport.cgi sun.cpu.endian : little sun.io.unicode.encoding : UnicodeLittle sun.desktop : gnome sun.cpu.isalist : ------------------------------------------- Temp dir ------------------------------------------- Temp dir is /tmp Temp dir is writeable Temp dir alignment with system clock is -555 ms ------------------------------------------- Locale information ------------------------------------------- Timezone Eastern Standard Time offset=-14400000 ------------------------------------------- Proxy information ------------------------------------------- Java1.5+ proxy settings: Direct connection
          Hide
          Steve Loughran added a comment -

          @fodil, run an ant -diagnostics on your command line and include the printout.

          Then try setting the HADOOP_OPTS property to whichever program you are trying to run

          Show
          Steve Loughran added a comment - @fodil, run an ant -diagnostics on your command line and include the printout. Then try setting the HADOOP_OPTS property to whichever program you are trying to run
          Hide
          fodil added a comment -

          1- I am using Hadoop 0.20.2

          2- Java -version returns
          java version "1.6.0_26"
          Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
          Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

          3- I have added the line : export HADOOP_OPTS="-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl" in the hadoop-env.sh

          the same problem.

          Any idea

          Show
          fodil added a comment - 1- I am using Hadoop 0.20.2 2- Java -version returns java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) 3- I have added the line : export HADOOP_OPTS="-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl" in the hadoop-env.sh the same problem. Any idea
          Hide
          Steve Loughran added a comment -
          1. Which version of Hadoop are you running? I'm not sure this is patched in the 0.20x branch.
          2. What does java -version say?
          3. It's a Java system property, so can be set in the environment variable HADOOP_OPTS, assuming you run the normal scripts. My deployment process will be different from yours, so there's no point me going in to more detail except to note that I patch HADOOP_OPTS in conf/hadoop-env.sh dynamically.
          Show
          Steve Loughran added a comment - Which version of Hadoop are you running? I'm not sure this is patched in the 0.20x branch. What does java -version say? It's a Java system property, so can be set in the environment variable HADOOP_OPTS, assuming you run the normal scripts. My deployment process will be different from yours, so there's no point me going in to more detail except to note that I patch HADOOP_OPTS in conf/hadoop-env.sh dynamically.
          Hide
          fodil added a comment -

          I have the same problem. adding the Putting xerces-2.9.1 and xml-apis-1.3.04 on the classpath did not resolve the problem.

          Here is the point of failure in the stack.

          Aug 4, 2011 6:34:32 AM org.apache.hadoop.conf.Configuration loadResource
          SEVERE: error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
          java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1171)
          at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1030)
          at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980)

          When you say to run the JVM with "-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl"

          Would tell me where can I specify this statement in order to force the JVM to use it?

          Thanks for your help

          Show
          fodil added a comment - I have the same problem. adding the Putting xerces-2.9.1 and xml-apis-1.3.04 on the classpath did not resolve the problem. Here is the point of failure in the stack. Aug 4, 2011 6:34:32 AM org.apache.hadoop.conf.Configuration loadResource SEVERE: error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1171) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1030) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980) When you say to run the JVM with "-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl" Would tell me where can I specify this statement in order to force the JVM to use it? Thanks for your help
          Hide
          steve_l added a comment -

          For people seeing this -it may be you have >1 parser on the classpath and the wrong one is being picked up

          run the JVm with

          -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
          
          Show
          steve_l added a comment - For people seeing this -it may be you have >1 parser on the classpath and the wrong one is being picked up run the JVm with -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
          Hide
          steve_l added a comment -

          The extra diagnostics flag up that some XML parsers (xerces 2.6.2?) can wait until a document is created before bailing out; this could be a sign of factory problems rather than just XML parser support issues; looking at the CP there are two Xerces versions there (2.6.2. and 2.9.1); clearly the wrong one is being used.

          09/03/13 12:01:04 [JobTracker] FATAL conf.Configuration : Error parsing configuration resource core-default.xml :
           javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
          javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
                  at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
                  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1063)
                  at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1032)
                  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:982)
                  at org.apache.hadoop.conf.Configuration.get(Configuration.java:384)
                  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:805)
                  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:835)
                  at org.apache.hadoop.mapred.JobTracker.getInstrumentationClass(JobTracker.java:1633)
                  at org.apache.hadoop.mapred.JobTracker.innerStart(JobTracker.java:1472)
                  at org.apache.hadoop.util.Service.start(Service.java:186)
          
          1. We could check in the extra diagnostics here -stack trace and file at fault; this will help others
          2. We could have a catch for a ParserConfigurationException and include an error message that points the viewer at a wiki page where we point to this bugrep and anything else relevant
          Show
          steve_l added a comment - The extra diagnostics flag up that some XML parsers (xerces 2.6.2?) can wait until a document is created before bailing out; this could be a sign of factory problems rather than just XML parser support issues; looking at the CP there are two Xerces versions there (2.6.2. and 2.9.1); clearly the wrong one is being used. 09/03/13 12:01:04 [JobTracker] FATAL conf.Configuration : Error parsing configuration resource core- default .xml : javax.xml.parsers.ParserConfigurationException: Feature 'http: //apache.org/xml/features/xinclude' is not recognized. javax.xml.parsers.ParserConfigurationException: Feature 'http: //apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1063) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1032) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:982) at org.apache.hadoop.conf.Configuration.get(Configuration.java:384) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:805) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:835) at org.apache.hadoop.mapred.JobTracker.getInstrumentationClass(JobTracker.java:1633) at org.apache.hadoop.mapred.JobTracker.innerStart(JobTracker.java:1472) at org.apache.hadoop.util.Service.start(Service.java:186) We could check in the extra diagnostics here -stack trace and file at fault; this will help others We could have a catch for a ParserConfigurationException and include an error message that points the viewer at a wiki page where we point to this bugrep and anything else relevant
          Hide
          steve_l added a comment -

          Seeing this again on a job tracker running in a VM

          09/03/12 16:25:20 [JobTracker] FATAL conf.Configuration : error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
          

          Adding more stack trace reporting to these errors (And the resource name) for better diagnostics; it may be some XML parsers throw a different exception when they encounter XInclude problems

          Show
          steve_l added a comment - Seeing this again on a job tracker running in a VM 09/03/12 16:25:20 [JobTracker] FATAL conf.Configuration : error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http: //apache.org/xml/features/xinclude' is not recognized. Adding more stack trace reporting to these errors (And the resource name) for better diagnostics; it may be some XML parsers throw a different exception when they encounter XInclude problems
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #758 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/758/)
          . The Configuration class should be able to work with XML
          parsers that do not support xmlinclude. (Steve Loughran via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #758 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/758/ ) . The Configuration class should be able to work with XML parsers that do not support xmlinclude. (Steve Loughran via dhruba)
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Steve!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Steve!
          Hide
          steve_l added a comment -

          failing test was org.apache.hadoop.mapred.TestJobInProgress.testRunningTaskCount , which timed out

          Show
          steve_l added a comment - failing test was org.apache.hadoop.mapred.TestJobInProgress.testRunningTaskCount , which timed out
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12400189/hadoop-5254.patch
          against trunk revision 744406.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400189/hadoop-5254.patch against trunk revision 744406. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3862/console This message is automatically generated.
          Hide
          steve_l added a comment -

          I've only done it against trunk; not looked at 0.20.

          It would be nice to have some details on what triggers the problem; which JVMs do/dont exhibit this problem.

          1. all the hadoop core tests worked
          2. my single VM tests worked
          3. it was the stuff trying to deploy and test onto a VM-ware hosted image that caused problems.
          4. Once I pushed xerces2.9.1 onto the classpath, things went away

          I think the hadoop core tests aren't noticing this because Ant ships with xerces; junit-under-ant gets tht when you tell junit to inherit Ant's classpath.
          My VM tests may have been working for the same reason.
          It was only the stuff running standalone that was in trouble.

          I'll ask around -what we need is a table that shows JDK and Xerces versions with features in each

          Show
          steve_l added a comment - I've only done it against trunk; not looked at 0.20. It would be nice to have some details on what triggers the problem; which JVMs do/dont exhibit this problem. all the hadoop core tests worked my single VM tests worked it was the stuff trying to deploy and test onto a VM-ware hosted image that caused problems. Once I pushed xerces2.9.1 onto the classpath, things went away I think the hadoop core tests aren't noticing this because Ant ships with xerces; junit-under-ant gets tht when you tell junit to inherit Ant's classpath. My VM tests may have been working for the same reason. It was only the stuff running standalone that was in trouble. I'll ask around -what we need is a table that shows JDK and Xerces versions with features in each
          Hide
          dhruba borthakur added a comment -

          +1. This code looks good. Do you intend this for trunk as well as 0.20?

          Show
          dhruba borthakur added a comment - +1. This code looks good. Do you intend this for trunk as well as 0.20?
          Hide
          steve_l added a comment -

          patch to log and continue. retains original behaviour (no Xinclude) on systems whose parsers don't have it

          Show
          steve_l added a comment - patch to log and continue. retains original behaviour (no Xinclude) on systems whose parsers don't have it
          Hide
          steve_l added a comment -

          Putting xerces-2.9.1 and xml-apis-1.3.04 on the classpath appears to eliminate this problem.

          Which implies two actions

          1. Require XInclude in the Parser
          • have some explicit restrictions on which XML parsers to use, knowing that some setups won't have it
          • consider bundling Xerces/xml-apis with Hadoop to reduce support calls
          • doc that all programs that need to load a Configuration() instance had better have it
          • catch and rethrow the exception, this time with a helpful error message pointing to the relevant web page
          1. Ignore XInclude when not enabled
          • catch and log the exception
          • backwards compatible
          • will cause problems if people want XInclude, and don't get it.
          • may cause some tests to fail (not seen that myself yet)
          • consider recognising XInclude errors and having more meaningful error messages "upgrade your parser"
          Show
          steve_l added a comment - Putting xerces-2.9.1 and xml-apis-1.3.04 on the classpath appears to eliminate this problem. Which implies two actions Require XInclude in the Parser have some explicit restrictions on which XML parsers to use, knowing that some setups won't have it consider bundling Xerces/xml-apis with Hadoop to reduce support calls doc that all programs that need to load a Configuration() instance had better have it catch and rethrow the exception, this time with a helpful error message pointing to the relevant web page Ignore XInclude when not enabled catch and log the exception backwards compatible will cause problems if people want XInclude, and don't get it. may cause some tests to fail (not seen that myself yet) consider recognising XInclude errors and having more meaningful error messages "upgrade your parser"
          Hide
          steve_l added a comment -

          This wraps the failure, logs it and continues. While it stops XInclude working on those runtimes without XInclude support, it ensures that people can still run Hadoop on Java 6 versions that are not XInclude enabled.

          Show
          steve_l added a comment - This wraps the failure, logs it and continues. While it stops XInclude working on those runtimes without XInclude support, it ensures that people can still run Hadoop on Java 6 versions that are not XInclude enabled.
          Hide
          steve_l added a comment -

          the root cause is in the stack trace, the base class impl of setXIncludeAware() is (and this is sun code, so no cut and paste in apache):-

              public void setXIncludeAware(final boolean state) {
                  throw new UnsupportedOperationException(
                      "This parser does not support specification \""
                      + this.getClass().getPackage().getSpecificationTitle()
                      + "\" version \""
                      + this.getClass().getPackage().getSpecificationVersion()
                      + "\""
                      );
              }
          

          Proposed: catch and log@error, add some XML parser diags.

          Incidentally, my JVM version is JRockit. I wonder if it is because it is pre 1.6u5, and that when Sun pushed out new features in that update, it included XInclude awareness -as some versions of Java6 clearly do not have it

          Show
          steve_l added a comment - the root cause is in the stack trace, the base class impl of setXIncludeAware() is (and this is sun code, so no cut and paste in apache):- public void setXIncludeAware( final boolean state) { throw new UnsupportedOperationException( "This parser does not support specification \" " + this .getClass().getPackage().getSpecificationTitle() + "\" version \"" + this .getClass().getPackage().getSpecificationVersion() + "\" " ); } Proposed: catch and log@error, add some XML parser diags. Incidentally, my JVM version is JRockit. I wonder if it is because it is pre 1.6u5, and that when Sun pushed out new features in that update, it included XInclude awareness -as some versions of Java6 clearly do not have it
          Hide
          steve_l added a comment -

          Assumed cause

          Show
          steve_l added a comment - Assumed cause
          Hide
          steve_l added a comment -

          This is the root cause of another test failure of mine, something that tried to load every class that namenode needed, one by one, and resources, and told me off if they were missing. Class.forName("org.apache.hadoop.hdfs.protocol.FSConstants") was throwing an CNFE, but the .class file was around.

          Looks like an ExceptionInInitializerError gets turned into a ClassNotFoundException in Class.forName(), without any attempt to nest in the underlying exception. Because that woudl be too helpful.

          Caused by: java.lang.ClassNotFoundException: forName: loading of org.apache.hadoop.hdfs.protocol.FSConstants failed, but loading it without references succeeded. Check references.
          at org.smartfrog.sfcore.security.SFClassLoader.forName(SFClassLoader.java:570)
          at org.smartfrog.services.os.java.LoadClassImpl.loadClasses(LoadClassImpl.java:160)
          at org.smartfrog.services.os.java.LoadClassImpl.loadResourcesAndClasses(LoadClassImpl.java:118)
          at org.smartfrog.services.os.java.LoadClassImpl.sfStart(LoadClassImpl.java:104)
          at org.smartfrog.services.assertions.TestCompoundImpl.sfStart(TestCompoundImpl.java:248)
          at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
          at sun.rmi.transport.Transport$1.run(Transport.java:159)

          Show
          steve_l added a comment - This is the root cause of another test failure of mine, something that tried to load every class that namenode needed, one by one, and resources, and told me off if they were missing. Class.forName("org.apache.hadoop.hdfs.protocol.FSConstants") was throwing an CNFE, but the .class file was around. Looks like an ExceptionInInitializerError gets turned into a ClassNotFoundException in Class.forName(), without any attempt to nest in the underlying exception. Because that woudl be too helpful. Caused by: java.lang.ClassNotFoundException: forName: loading of org.apache.hadoop.hdfs.protocol.FSConstants failed, but loading it without references succeeded. Check references. at org.smartfrog.sfcore.security.SFClassLoader.forName(SFClassLoader.java:570) at org.smartfrog.services.os.java.LoadClassImpl.loadClasses(LoadClassImpl.java:160) at org.smartfrog.services.os.java.LoadClassImpl.loadResourcesAndClasses(LoadClassImpl.java:118) at org.smartfrog.services.os.java.LoadClassImpl.sfStart(LoadClassImpl.java:104) at org.smartfrog.services.assertions.TestCompoundImpl.sfStart(TestCompoundImpl.java:248) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159)
          Hide
          steve_l added a comment -

          Full stack trace

          full stack trace

          [sf-startdaemon-debug] java.lang.ExceptionInInitializerError
          [sf-startdaemon-debug] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:76)
          [sf-startdaemon-debug] at java.lang.Thread.run(Thread.java:619)
          [sf-startdaemon-debug] Caused by: java.lang.UnsupportedOperationException: This parser does not support specification "null" version "null"
          [sf-startdaemon-debug] at javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
          [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1052)
          [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1029)
          [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979)
          [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.get(Configuration.java:381)
          [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:450)
          [sf-startdaemon-debug] at org.apache.hadoop.hdfs.protocol.FSConstants.<clinit>(FSConstants.java:51)
          [sf-startdaemon-debug] ... 2 more

          Show
          steve_l added a comment - Full stack trace full stack trace [sf-startdaemon-debug] java.lang.ExceptionInInitializerError [sf-startdaemon-debug] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:76) [sf-startdaemon-debug] at java.lang.Thread.run(Thread.java:619) [sf-startdaemon-debug] Caused by: java.lang.UnsupportedOperationException: This parser does not support specification "null" version "null" [sf-startdaemon-debug] at javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590) [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1052) [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1029) [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979) [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.get(Configuration.java:381) [sf-startdaemon-debug] at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:450) [sf-startdaemon-debug] at org.apache.hadoop.hdfs.protocol.FSConstants.<clinit>(FSConstants.java:51) [sf-startdaemon-debug] ... 2 more

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development