[GEODE-1170] An incorrect gateway-sender element for an AsyncEventQueue is generated in the xml during reconnect causing a GatewaySenderException when the member rejoins

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0-incubating.M2
• Component/s: wan
• Assignee: Barrett Oglesby
• Reporter: Barrett Oglesby
• Created: 05/Apr/16 00:14
• Updated: 12/Apr/16 05:59
• Resolved: 06/Apr/16 17:08

Description

If a Cache configures an AsyncEventQueue, and the member is forced out of the DistributedSystem, an incorrect gateway-sender element is generated in the xml to rebuild the cache after reconnect completes.

Original xml:

<gateway-sender id="ny" remote-distributed-system-id="1" ... />
<async-event-queue id="db" ... >

Generated xml:

[info 2016/04/04 15:04:29.979 PDT gateway-ln-1 <ServerConnection on port 5212 Thread 0> tid=0x66] generating XML to rebuild the cache after reconnect completes

[info 2016/04/04 15:04:30.014 PDT gateway-ln-1 <ServerConnection on port 5212 Thread 0> tid=0x66] XML generation completed: <?xml version="1.0" encoding="UTF-8"?><cache ... >
  <gateway-sender id="ny" remote-distributed-system-id="1" ... />
  <gateway-sender id="AsyncEventQueue_db" remote-distributed-system-id="-1" ... />
  <async-event-queue id="db" ... >

Exception during reconnect:

[warning 2016/04/04 15:05:30.711 PDT gateway-ln-1 <ReconnectThread> tid=0x68] Exception occurred while trying to create the cache during reconnect
com.gemstone.gemfire.cache.CacheXmlException: While reading Cache XML file:/path/to/gemfire-ln-hub.xml. While parsing XML, caused by com.gemstone.gemfire.internal.cache.wan.GatewaySenderException: GatewaySender AsyncEventQueue_db cannot be created with remote DS Id less than 0. 
	at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.parse(CacheXmlParser.java:275)
	at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4384)
	at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1364)
	at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1176)
	at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:796)
	at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:780)
	at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2814)
	at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2526)
	at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:947)
	at com.gemstone.gemfire.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4381)
	at com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1580)
	at com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager$5.run(GMSMembershipManager.java:2615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.gemstone.gemfire.internal.cache.wan.GatewaySenderException: GatewaySender AsyncEventQueue_db cannot be created with remote DS Id less than 0. 
	at com.gemstone.gemfire.internal.cache.wan.GatewaySenderFactoryImpl.create(GatewaySenderFactoryImpl.java:199)
	at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.endSerialGatewaySender(CacheXmlParser.java:830)
	at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.endElement(CacheXmlParser.java:3166)
	at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser$DefaultHandlerDelegate.endElement(CacheXmlParser.java:3726)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
	at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:183)
	at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.emptyElement(XMLSchemaValidator.java:780)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:356)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2786)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
	at com.gemstone.gemfire.internal.cache.xmlcache.CacheXmlParser.parse(CacheXmlParser.java:230)
	... 12 more

The issue is that AsyncEventQueueFactoryImpl.create no longer uses a GatewaySenderFactory to create a GatewaySender. Instead, the GatewaySender is created directly from the attributes defined in the AsyncEventQueueFactoryImpl. Those attributes don't set isForInternalUse=true, so the resulting GatewaySender is not created as an internal one. Later, when CacheXmlGenerator.parse asks the Cache for its GatewaySenders, it gets the internal one used for the AsyncEventQueue and generates xml for it.

The fix is to set isForInternalUse=true in AsyncEventQueueFactoryImpl.configureAsyncEventQueue like:

this.attrs.isForInternalUse = true;

Generated xml with fix:

[info 2016/04/04 16:46:57.476 PDT gateway-ln-1 <ServerConnection on port 5456 Thread 0> tid=0x66] XML generation completed: <?xml version="1.0" encoding="UTF-8"?><cache ... >
  <gateway-sender id="ny" remote-distributed-system-id="1" ... />
  <async-event-queue id="db" ... >