[GEODE-8200] Rebalance operations stuck in "IN_PROGRESS" state forever - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 1.14.0, 1.15.0
Fix Version/s: 1.15.0
Component/s: management
Labels:

Description

We use the management REST API to call rebalance immediately before stopping a server to limit the possibility of data loss. In a cluster with 3 locators, 3 servers, and no regions, we noticed that sometimes the rebalance operation never ends if one of the locators is restarting concurrently with the rebalance operation.

More specifically, the scenario where we see this issue crop up is during an automated "rolling restart" operation in a Kubernetes environment which proceeds as follows:

At most one locator and one server are restarting at any point in time
Each locator/server waits until the previous locator/server is fully online before restarting
Immediately before stopping a server, a rebalance operation is performed and the server is not stopped until the rebalance operation is completed

The impact of this issue is that the "rolling restart" operation will never complete, because it cannot proceed with stopping a server until the rebalance operation is completed. A human is then required to intervene and manually trigger a rebalance and stop the server. This type of "rolling restart" operation is triggered fairly often in Kubernetes — any time part of the configuration of the locators or servers changes.

The following JSON is a sample response from the management REST API that shows the rebalance operation stuck in "IN_PROGRESS".

    {
      "statusCode": "IN_PROGRESS",
      "links": {
        "self": "http://geodecluster-sample-locator.default/management/v1/operations/rebalances/a47f23c8-02b3-443c-a367-636fd6921ea7",
        "list": "http://geodecluster-sample-locator.default/management/v1/operations/rebalances"
      },
      "operationStart": "2020-05-27T22:38:30.619Z",
      "operationId": "a47f23c8-02b3-443c-a367-636fd6921ea7",
      "operation": {
        "simulate": false
      }
    }