Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-7945

Cluster restart recovery from disk blocked by waiting replies of CreateRegionMessage

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.13.0
    • None

    Description

      A cluster restart recovering from disk has unexpected delays in some of the members. The logs show that those delayed members are waiting for replies of CreateRegionMessage, before loading the krf files. And the reason for waiting the replies of CreateRegionMessage is likely because other members are holding some lock while busy loading the krf files. Therefore the replies are delayed.

      Once those delayed members get the replies of CreateRegionMessage, it starts loading the krf files. If the delayed members contain the latest data, this could block other members waiting for the latest data.

      Because the cluster members are blocking each other at different stages during the cluster restart recovery process, the whole process is unexpectedly long when the disk store contains large amount of data.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jchen21 Jianxia Chen
            jchen21 Jianxia Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1.5h
                1.5h

                Slack

                  Issue deployment