[IGNITE-8053] Exception during checkpoint concurrent changes in topology

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4
• Fix Version/s: 2.5
• Component/s: persistence
• Assignee: Alexey Goncharuk
• Reporter: Sergey Kosarev
• Created: 27/Mar/18 07:31
• Updated: 30/Mar/18 14:57
• Resolved: 30/Mar/18 14:57

Description

Excpetion in checkpoint thread

2018-03-23 15:25:19.085 [ERROR][db-checkpoint-thread-#256%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.p.GridCacheDatabaseSharedManager] Runtime error caught during grid runnable execution: GridWorker [name=
db-checkpoint-thread, igniteInstanceName=DPL_GRID%DplGridNodeName, finished=false, hashCode=981946370, interrupted=false, runner=db-checkpoint-thread-#256%DPL_GRID%DplGridNodeName%]
java.lang.IllegalStateException: Failed to add new partition to the partitions state (no enough space reserved) [partId=32321, reserved=5416]
  at org.apache.ignite.internal.pagemem.wal.record.CacheState.addPartitionState(CacheState.java:64)
  at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:2955)
  at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:2704)
  at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:2629)
  at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
  at java.lang.Thread.run(Thread.java:745)

threre are 2 sequential invocations grp.topology().currentLocalPartitions() in
GridCacheDatabaseSharedManager.Checkpointer#markCheckpointBegin
it's assumed that results must be equal, but they doesn't actually.

Possible problem is in
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl#forceCreatePartition
it does not check for checkpoint lock
ctx.database().checkpointLockIsHeldByThread();