Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-6975

When a redundant copy or replica of a distributed region failed to persistent remote member's new persistence id, it should send reply exception back to indicate what happened

    XMLWordPrintableJSON

    Details

      Description

      Currently, when a persistent bucket or distributed region is created on member A, member A will send its new PersistentMemberID to other hosts (e.g member B), so that member B will know and persist A's new ID for the region.

      However, when member B is being shut down during processing the PrepareNewPersistentMemberMessage (did not persist A's id), it just send a reply message indicate it had persisted. This will cause Member A removes its old member id and only persists its new member id. This is wrong as the member A could also been shut down at the same time. There is a race that member B could be recognized as hosting the last copy for the region. This will lead to member B to recover first, and member B can only recover member A's old persistent id. This will lead to Member A not able to restart, as B does not recognize A's new persistent id.

      [error 2018/09/19 01:18:00.972 PDT dataStoregemfire6_host1_6131 <Recovery thread for bucket _B__partitionedRegion_0> tid=0x77] A DiskAccessException has occurred while writing to the disk for region /_PR/_B_partitionedRegion_0. The cache will be closed.
      org.apache.geode.cache.persistence.ConflictingPersistentDataException: Region /_PR/_B_partitionedRegion_0 remote member rs-FullRegression19041704a3i3large-hydra-client-62(dataStoregemfire1_host1_5862:5862)<ec><v8>:1025 with persistent data /10.32.109.230:/var/vcap/data/rundir/concParRegHAPersistPdxVA57H/concParRegHAPersistPdx-0919-011540/vm_1_dataStore1_disk_1 created at timestamp 1537345060760 version 0 diskStoreId a35a937a082b4066-af019365b6a5114b name null was not part of the same distributed system as the local data from /10.32.109.230:/var/vcap/data/rundir/concParRegHAPersistPdxVA57H/concParRegHAPersistPdx-0919-011540/vm_6_dataStore6_disk_1 created at timestamp 1537344996470 version 0 diskStoreId 108be5a03966418f-980c1d88e9b26d1d name null
      at org.apache.geode.internal.cache.persistence.PersistenceAdvisorImpl.checkMyStateOnMembers(PersistenceAdvisorImpl.java:521)
      at org.apache.geode.internal.cache.persistence.PersistenceInitialImageAdvisor.removeReplicatesIfWeAreEqualToAnyOrElseClearEqualMembers(PersistenceInitialImageAdvisor.java:181)
      at org.apache.geode.internal.cache.persistence.PersistenceInitialImageAdvisor.getAdvice(PersistenceInitialImageAdvisor.java:69)
      at org.apache.geode.internal.cache.persistence.PersistenceAdvisorImpl.getInitialImageAdvice(PersistenceAdvisorImpl.java:831)
      at org.apache.geode.internal.cache.persistence.CreatePersistentRegionProcessor.getInitialImageAdvice(CreatePersistentRegionProcessor.java:52)
      at org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1200)
      at org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1081)
      at org.apache.geode.internal.cache.BucketRegion.initialize(BucketRegion.java:258)
      at org.apache.geode.internal.cache.LocalRegion.createSubregion(LocalRegion.java:1014)
      at org.apache.geode.internal.cache.PartitionedRegionDataStore.createBucketRegion(PartitionedRegionDataStore.java:779)
      at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucket(PartitionedRegionDataStore.java:454)
      at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabBucket(PartitionedRegionDataStore.java:2895)
      at org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDisk(ProxyBucketRegion.java:447)
      at org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDiskRecursively(ProxyBucketRegion.java:390)
      at org.apache.geode.internal.cache.PRHARedundancyProvider$4.run2(PRHARedundancyProvider.java:1756)
      at org.apache.geode.internal.cache.partitioned.RecoveryRunnable.run(RecoveryRunnable.java:58)
      at org.apache.geode.internal.cache.PRHARedundancyProvider$4.run(PRHARedundancyProvider.java:1748)
      at java.lang.Thread.run(Thread.java:748)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                eshu Eric Shu
                Reporter:
                eshu Eric Shu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m