Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
We're seeing another data loss issue when upgrading servers due to GEODE-2137. The issue is that we store HAEventWrapper objects in a region for server->client queues. These objects contain a member ID. Because the objects are region values, they can be lazily deserialized using the version of the current member, which may not match the version of the member they were serialized with.
What we are seeing is that when a client is creating a queue on a 9.1 server, and it is copying the contents of the queue from a 9.0 server, we get are getting serialization errors which prevent the queue from being created. When the 9.0 server is killed as part of a rolling upgrade, this results in event loss.
[severe 2017/07/06 15:09:52.195 PDT <Handshaker 0.0.0.0/0.0.0.0:23779 Thread 6> tid=0xc0] Uncaught exception in thread Thread[Handshaker 0.0.0.0/0.0.0.0:23779 Thread 6,5,Handshaker 0.0.0.0/0.0.0.0:23779] org.apache.geode.InternalGemFireError: unexpected typeCode: -126 at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.InternalDataSerializer.decodePrimitiveClass(InternalDataSerializer.java:1880) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.DataSerializer.readClass(DataSerializer.java:264) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2707) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.tier.sockets.HAEventWrapper.fromData(HAEventWrapper.java:330) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.InternalDataSerializer.invokeFromData(InternalDataSerializer.java:2370) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.DSFIDFactory.create(DSFIDFactory.java:981) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2691) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.AbstractRegionMap.initialImagePut(AbstractRegionMap.java:771) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.InitialImageOperation.processChunk(InitialImageOperation.java:928) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.InitialImageOperation$ImageProcessor.process(InitialImageOperation.java:1249) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.distributed.internal.ReplyMessage.process(ReplyMessage.java:213) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.internal.cache.InitialImageOperation$ImageReplyMessage.process(InitialImageOperation.java:2723) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.distributed.internal.ReplyMessage.dmProcess(ReplyMessage.java:193) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.distributed.internal.ReplyMessage.process(ReplyMessage.java:186) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440) at Remote Member '172.1.1.1(32406)<v4>:32771' in java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at Remote Member '172.1.1.1(32406)<v4>:32771' in java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:665) at Remote Member '172.1.1.1(32406)<v4>:32771' in org.apache.geode.distributed.internal.DistributionManager$5$1.run(DistributionManager.java:961) at Remote Member '172.1.1.1(32406)<v4>:32771' in java.lang.Thread.run(Thread.java:748) at org.apache.geode.distributed.internal.ReplyException.handleAsUnexpected(ReplyException.java:89) at org.apache.geode.internal.cache.InitialImageOperation.getFromOne(InitialImageOperation.java:501) at org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1182) at org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1022) at org.apache.geode.internal.cache.HARegion.initialize(HARegion.java:337) at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3077) at org.apache.geode.internal.cache.HARegion.getInstance(HARegion.java:257) at org.apache.geode.internal.cache.ha.HARegionQueue.createHARegion(HARegionQueue.java:367) at org.apache.geode.internal.cache.ha.HARegionQueue.<init>(HARegionQueue.java:347) at org.apache.geode.internal.cache.ha.HARegionQueue$BlockingHARegionQueue.<init>(HARegionQueue.java:2100) at org.apache.geode.internal.cache.ha.HARegionQueue$DurableHARegionQueue.<init>(HARegionQueue.java:2331) at org.apache.geode.internal.cache.ha.HARegionQueue.getHARegionQueueInstance(HARegionQueue.java:1953) at org.apache.geode.internal.cache.tier.sockets.CacheClientProxy$MessageDispatcher.<init>(CacheClientProxy.java:2321) at org.apache.geode.internal.cache.tier.sockets.CacheClientProxy.initializeMessageDispatcher(CacheClientProxy.java:1730) at org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.initializeProxy(CacheClientNotifier.java:610) at org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.registerClient(CacheClientNotifier.java:537) at org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.registerGFEClient(CacheClientNotifier.java:330) at org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.registerClient(CacheClientNotifier.java:277) at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl.handleNewClientConnection(AcceptorImpl.java:1457) at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$5.run(AcceptorImpl.java:1332) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)
Attachments
Issue Links
- links to
