[GEODE-8827] The DiskRegionStats bytesOnlyOnDisk stat is not incremented during persistent region recovery

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: persistence, statistics
• Labels: pull-request-available
• Assignee: Barrett Oglesby
• Reporter: Barrett Oglesby
• Created: 12/Jan/21 19:50
• Updated: 03/Sep/21 02:22
• Resolved: 20/Jan/21 20:07

Description

With a test like:

• 2 servers with partitioned region configured like:
  • persistence enabled
  • heap eviction with overflow enabled
• load enough entries to cause overflow
• shut down the servers
• restart the servers
• execute a function to get all entries in each server

After the step to restart the servers, the bytesOnlyOnDisk stat is 0.

After the step to get all entries, the bytesOnlyOnDisk stat is negative.

The entriesInVM and entriesOnlyOnDisk stats are incremented as BucketRegions are recovered from disk in LocalRegion.initializeStats here:

java.lang.Exception: Stack trace
	at java.lang.Thread.dumpStack(Thread.java:1333)
	at org.apache.geode.internal.cache.LocalRegion.initializeStats(LocalRegion.java:10222)
	at org.apache.geode.internal.cache.BucketRegion.initializeStats(BucketRegion.java:2163)
	at org.apache.geode.internal.cache.AbstractDiskRegion.copyExistingRegionMap(AbstractDiskRegion.java:775)
	at org.apache.geode.internal.cache.DiskStoreImpl.initializeOwner(DiskStoreImpl.java:631)
	at org.apache.geode.internal.cache.DiskRegion.initializeOwner(DiskRegion.java:239)
	at org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1081)
	at org.apache.geode.internal.cache.BucketRegion.initialize(BucketRegion.java:262)
	at org.apache.geode.internal.cache.LocalRegion.createSubregion(LocalRegion.java:981)
	at org.apache.geode.internal.cache.PartitionedRegionDataStore.createBucketRegion(PartitionedRegionDataStore.java:785)
	at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucket(PartitionedRegionDataStore.java:460)
	at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucketRecursively(PartitionedRegionDataStore.java:319)
	at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabBucket(PartitionedRegionDataStore.java:2896)
	at org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDisk(ProxyBucketRegion.java:441)
	at org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDiskRecursively(ProxyBucketRegion.java:407)
	at org.apache.geode.internal.cache.PRHARedundancyProvider$2.run2(PRHARedundancyProvider.java:1640)
	at org.apache.geode.internal.cache.partitioned.RecoveryRunnable.run(RecoveryRunnable.java:60)
	at org.apache.geode.internal.cache.PRHARedundancyProvider$2.run(PRHARedundancyProvider.java:1630)
	at java.lang.Thread.run(Thread.java:745)

The current LocalRegion.initializeStats method implementation is:

public void initializeStats(long numEntriesInVM, long numOverflowOnDisk,
    long numOverflowBytesOnDisk) {
  getDiskRegion().getStats().incNumEntriesInVM(numEntriesInVM);
  getDiskRegion().getStats().incNumOverflowOnDisk(numOverflowOnDisk);
}

Even though numOverflowBytesOnDisk is passed into this method, it is ignored as this logging shows:

[warn 2021/01/12 11:19:11.785 PST  <Recovery thread for bucket _B__data_3> tid=0x49] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4546560; bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.791 PST  <Recovery thread for bucket _B__data_9> tid=0x4f] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4536320; bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.797 PST  <Recovery thread for bucket _B__data_6> tid=0x4c] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4526080; bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.800 PST  <Recovery thread for bucket _B__data_2> tid=0x48] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4546560; bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.801 PST  <Recovery thread for bucket _B__data_8> tid=0x4e] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4526080; bytesOnlyOnDiskFromStats=0

The attached DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_restart.gif chart shows the behavior after server restart.

The attached DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_gets.gif chart shows the behavior after the gets.

If I make this change, I see correct behavior:

  public void initializeStats(long numEntriesInVM, long numOverflowOnDisk,
      long numOverflowBytesOnDisk) {
    getDiskRegion().getStats().incNumEntriesInVM(numEntriesInVM);
    getDiskRegion().getStats().incNumOverflowOnDisk(numOverflowOnDisk);
->  getDiskRegion().getStats().incNumOverflowBytesOnDisk(numOverflowBytesOnDisk);
  }

The attached DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_restart_with_change.gif chart shows the behavior with this change after server restart.

The attached DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_gets_with_change.gif chart shows the behavior with this change after the gets.

Issue Links

• links to: GitHub Pull Request #5898