Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-8200

Rebalance operations stuck in "IN_PROGRESS" state forever

    XMLWordPrintableJSON

    Details

      Description

      We use the management REST API to call rebalance immediately before stopping a server to limit the possibility of data loss. In a cluster with 3 locators, 3 servers, and no regions, we noticed that sometimes the rebalance operation never ends if one of the locators is restarting concurrently with the rebalance operation.

      More specifically, the scenario where we see this issue crop up is during an automated "rolling restart" operation in a Kubernetes environment which proceeds as follows:

      • At most one locator and one server are restarting at any point in time
      • Each locator/server waits until the previous locator/server is fully online before restarting
      • Immediately before stopping a server, a rebalance operation is performed and the server is not stopped until the rebalance operation is completed

      The impact of this issue is that the "rolling restart" operation will never complete, because it cannot proceed with stopping a server until the rebalance operation is completed. A human is then required to intervene and manually trigger a rebalance and stop the server. This type of "rolling restart" operation is triggered fairly often in Kubernetes — any time part of the configuration of the locators or servers changes.

      The following JSON is a sample response from the management REST API that shows the rebalance operation stuck in "IN_PROGRESS".

          {
            "statusCode": "IN_PROGRESS",
            "links": {
              "self": "http://geodecluster-sample-locator.default/management/v1/operations/rebalances/a47f23c8-02b3-443c-a367-636fd6921ea7",
              "list": "http://geodecluster-sample-locator.default/management/v1/operations/rebalances"
            },
            "operationStart": "2020-05-27T22:38:30.619Z",
            "operationId": "a47f23c8-02b3-443c-a367-636fd6921ea7",
            "operation": {
              "simulate": false
            }
          }
      

        Attachments

        1. GEODE-8200-exportedLogs.zip
          247 kB
          Aaron Lindsey

          Activity

            People

            • Assignee:
              jchen21 Jianxia Chen
              Reporter:
              aaronlindsey Aaron Lindsey
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: