[HBASE-18282] ReplicationLogCleaner can delete WALs not yet replicated in case of a KeeperException

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1
• Fix Version/s: 1.3.2, 1.4.2, 2.0.0, 1.2.7
• Component/s: Replication
• Assignee: Ben Lau
• Reporter: Ashu Pachauri
• Created: 27/Jun/17 23:10
• Updated: 01/Feb/19 19:55
• Resolved: 15/Feb/18 02:36

Description

ReplicationStateZKBase#getListOfReplicators does not rethrow a KeeperException and returns null in such a case. ReplicationLogCleaner just assumes that there are no replicators and deletes everything.

ReplicationStateZKBase:

public List<String> getListOfReplicators() {
    List<String> result = null;
    try {
      result = ZKUtil.listChildrenNoWatch(this.zookeeper, this.queuesZNode);
    } catch (KeeperException e) {
      this.abortable.abort("Failed to get list of replicators", e);
    }
    return result;
  }

ReplicationLogCleaner:

private Set<String> loadWALsFromQueues() throws KeeperException {
    for (int retry = 0; ; retry++) {
      int v0 = replicationQueues.getQueuesZNodeCversion();
      List<String> rss = replicationQueues.getListOfReplicators();
      if (rss == null) {
        LOG.debug("Didn't find any region server that replicates, won't prevent any deletions.");
        return ImmutableSet.of();
      }
      ...