Major Description
A test failed and the logs were found to be full of NPEs from the tombstone service:[severe 2016/12/20 02:04:35.605 UTC dataStoregemfire7_rs-StorageBTTest-2016-12-19-23-35-42-client-14_19508 <Replicate/Partition Region Garbage Collector> tid=0x44] GemFire garbage collection service encountered an unexpected exception
java.lang.NullPointerException
at org.apache.geode.internal.cache.TombstoneService$TombstoneSweeper.lambda$purgeObsoleteTombstones$1(TombstoneService.java:938)
at org.apache.geode.internal.cache.TombstoneService$ReplicateTombstoneSweeper.removeExpiredIf(TombstoneService.java:479)
at org.apache.geode.internal.cache.TombstoneService$TombstoneSweeper.removeIf(TombstoneService.java:823)
at org.apache.geode.internal.cache.TombstoneService$TombstoneSweeper.purgeObsoleteTombstones(TombstoneService.java:937)
at org.apache.geode.internal.cache.TombstoneService$TombstoneSweeper.run(TombstoneService.java:880)
at java.lang.Thread.run(Thread.java:745)
[severe 2016/12/20 02:05:45.987 UTC dataStoregemfire7_rs-StorageBTTest-2016-12-19-23-35-42-client-14_19508 <Replicate/Partition Region Garbage Collector> tid=0x44] GemFire garbage collection service encountered an unexpected exception
java.lang.NullPointerException
at org.apache.geode.internal.cache.TombstoneService$ReplicateTombstoneSweeper.expireBatch(TombstoneService.java:524)
at org.apache.geode.internal.cache.TombstoneService$ReplicateTombstoneSweeper.checkExpiredTombstoneGC(TombstoneService.java:594)
at org.apache.geode.internal.cache.TombstoneService$TombstoneSweeper.run(TombstoneService.java:878)
at java.lang.Thread.run(Thread.java:745)
Both of these stacks indicate that the "expiredTombstones" ArrayList somehow has nulls in it. It is an ArrayList of Tombstone instances and the only code that adds to it first tests that the item it is adding is not null. The only other modify operation done on it is to remove an item.
Perhaps unsafe concurrent access is happening causing this code to see nulls.