[GEODE-7945] Cluster restart recovery from disk blocked by waiting replies of CreateRegionMessage

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Labels: GeodeCommons
• Assignee: Jianxia Chen
• Reporter: Jianxia Chen
• Created: 02/Apr/20 19:00
• Updated: 10/Sep/20 15:51
• Resolved: 09/Apr/20 00:10

Description

A cluster restart recovering from disk has unexpected delays in some of the members. The logs show that those delayed members are waiting for replies of CreateRegionMessage, before loading the krf files. And the reason for waiting the replies of CreateRegionMessage is likely because other members are holding some lock while busy loading the krf files. Therefore the replies are delayed.

Once those delayed members get the replies of CreateRegionMessage, it starts loading the krf files. If the delayed members contain the latest data, this could block other members waiting for the latest data.

Because the cluster members are blocking each other at different stages during the cluster restart recovery process, the whole process is unexpectedly long when the disk store contains large amount of data.

Issue Links

• links to: GitHub Pull Request #4899