Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9299 Node Blacklisting: Coordinators should blacklist unhealthy nodes
  3. IMPALA-9342

Membership updates should only remove quiescing nodes from the blacklist

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 3.4.0
    • Component/s: Backend
    • Labels:
      None
    • Epic Color:
      ghx-label-7

      Description

      ClusterMembershipMgr::UpdateMembership will remove a node from the blacklist (if it is on the blacklist) if the method receives an update from the Statestore about the node. Currently, the Statestore should only send an update about the node if the node starts quiescing. If a node starts quiescing, it should be removed from the blacklist since it quiescing nodes aren't part of any executor groups anyway (no queries should be scheduled on them).

      After running some experiments locally, it seems there are some other cases where the Statestore sends the ClusterMembershipMgr an update about a node even if it's quiescing state has not changed. Unfortunately, I haven't been able to fully track down what is triggering this, so far it only happens on cluster start up.

      The ClusterMembershipMgr should only un-blacklist a node if that node is quiescing, currently it un-blacklists a node on any update to the node.

        Attachments

          Activity

            People

            • Assignee:
              stakiar Sahil Takiar
              Reporter:
              stakiar Sahil Takiar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: