[AMBARI-21074] Storm XML File has Invalid Characters Causing Exceptions on Server Startup

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.5.1
• Fix Version/s: 2.5.1
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 19/May/17 14:19
• Updated: 04/Oct/19 16:12
• Resolved: 19/May/17 14:51

Description

Ambari server is failing to start on D7 due to multiple exceptions during ambari-server schema upgrade.
The first exception thrown is

19 May 2017 10:01:42,268  INFO [main] ServiceDirectory:419 - Role command order info was loaded from file: /var/lib/ambari-server/resources/common-services/DRUID/0.9.2/role_command_order.json
19 May 2017 10:01:42,434 ERROR [main] ModuleFileUnmarshaller:141 - Cannot parse /var/lib/ambari-server/resources/common-services/STORM/0.9.1/configuration/storm-env.xml
19 May 2017 10:01:42,435 ERROR [main] ModuleFileUnmarshaller:143 - The entity name must immediately follow the '&' in the entity reference.
org.xml.sax.SAXParseException; systemId: file:/var/lib/ambari-server/resources/common-services/STORM/0.9.1/configuration/storm-env.xml; lineNumber: 131; columnNumber: 64; The entity name must immediately follow the '&' in the entity reference.
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
        at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
        at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
        at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1846)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3061)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:216)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:189)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:157)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:162)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:171)
        at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:189)
        at org.apache.ambari.server.stack.ModuleFileUnmarshaller.unmarshal(ModuleFileUnmarshaller.java:136)
        at org.apache.ambari.server.stack.ModuleFileUnmarshaller.unmarshal(ModuleFileUnmarshaller.java:81)
        at org.apache.ambari.server.stack.ConfigurationDirectory.parsePath(ConfigurationDirectory.java:103)
        at org.apache.ambari.server.stack.ConfigurationDirectory.<init>(ConfigurationDirectory.java:79)
        at org.apache.ambari.server.stack.StackDefinitionDirectory.getConfigurationDirectory(StackDefinitionDirectory.java:72)
        at org.apache.ambari.server.stack.ServiceModule.populateConfigurationModules(ServiceModule.java:393)
        at org.apache.ambari.server.stack.ServiceModule.<init>(ServiceModule.java:151)
        at org.apache.ambari.server.stack.StackManager.parseCommonServicesDirectory(StackManager.java:514)
        at org.apache.ambari.server.stack.StackManager.parseDirectories(StackManager.java:186)
        at org.apache.ambari.server.stack.StackManager.<init>(StackManager.java:158)
        at org.apache.ambari.server.stack.StackManager$$FastClassByGuice$$33e4ffe0.newInstance(<generated>)

Then there are more guice errors

19 May 2017 10:01:44,916 ERROR [main] SchemaUpgradeHelper:240 - Upgrade failed.
com.google.inject.ProvisionException: Guice provision errors:

1) Error injecting method, java.lang.RuntimeException: Trying to create a ServiceComponent not recognized in stack info, clusterName=cl1, serviceName=STORM, componentName=DRPC_SERVER, stackInfo=HDF-2.1
  at org.apache.ambari.server.state.cluster.ClustersImpl.loadClustersAndHosts(ClustersImpl.java:186)
  at org.apache.ambari.server.state.cluster.ClustersImpl.class(ClustersImpl.java:102)
  while locating org.apache.ambari.server.state.cluster.ClustersImpl
  while locating org.apache.ambari.server.state.Clusters

1 error
        at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.addNewConfigurationsFromXml(AbstractUpgradeCatalog.java:367)
        at org.apache.ambari.server.upgrade.UpgradeCatalog250.executeDMLUpdates(UpgradeCatalog250.java:195)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:940)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:238)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:431)
Caused by: java.lang.RuntimeException: Trying to create a ServiceComponent not recognized in stack info, clusterName=cl1, serviceName=STORM, componentName=DRPC_SERVER, stackInfo=HDF-2.1
        at org.apache.ambari.server.state.ServiceComponentImpl.updateComponentInfo(ServiceComponentImpl.java:134)
        at org.apache.ambari.server.state.ServiceComponentImpl.<init>(ServiceComponentImpl.java:164)
        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:260)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
        at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
        at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
        at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
        at com.sun.proxy.$Proxy19.createExisting(Unknown Source)
        at org.apache.ambari.server.state.ServiceImpl.<init>(ServiceImpl.java:169)
        at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
        at com.google.inject.internal.ProxyFactory$ProxyConstructor.newInstance(ProxyFactory.java:260)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
        at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
        at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
        at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
        at com.sun.proxy.$Proxy15.createExisting(Unknown Source)

Cluster details

Cluster will be up for the next 48 hours. Use the attached keypair.

172.22.74.229	natd76-xems-ambarieu-hdf-212-to-300-sec-5.openstacklocal	natd76-xems-ambarieu-hdf-212-to-300-sec-5	natd76-xems-ambarieu-hdf-212-to-300-sec-5.openstacklocal.
172.22.74.208	natd76-xems-ambarieu-hdf-212-to-300-sec-4.openstacklocal	natd76-xems-ambarieu-hdf-212-to-300-sec-4	natd76-xems-ambarieu-hdf-212-to-300-sec-4.openstacklocal.
172.22.74.209	natd76-xems-ambarieu-hdf-212-to-300-sec-3.openstacklocal	natd76-xems-ambarieu-hdf-212-to-300-sec-3	natd76-xems-ambarieu-hdf-212-to-300-sec-3.openstacklocal.
172.22.74.207	natd76-xems-ambarieu-hdf-212-to-300-sec-2.openstacklocal	natd76-xems-ambarieu-hdf-212-to-300-sec-2	natd76-xems-ambarieu-hdf-212-to-300-sec-2.openstacklocal.
172.22.74.21	natd76-xems-ambarieu-hdf-212-to-300-sec-1.openstacklocal	natd76-xems-ambarieu-hdf-212-to-300-sec-1	natd76-xems-ambarieu-hdf-212-to-300-sec-1.openstacklocal.

Issue Links

• is duplicated by: AMBARI-21066 validatePropertySchemaOfServiceXMLs UT fails
• links to: Reviewboard