[MESOS-3790] ZooKeeper connection should retry on EAI_NONAME

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.5.2, 1.6.1, 1.7.0
• Labels: mesosphere zookeeper
• Assignee: Andrew Schwartzmeyer
• Reporter: Neil Conway
• Created: 22/Oct/15 04:34
• Updated: 26/Nov/18 12:22
• Resolved: 26/Nov/18 12:22

Description

The zookeeper interface is designed to retry (once per second for up to ten minutes) if one or more of the Zookeeper hostnames can't be resolved (see MESOS-1326 and MESOS-1523).

However, the current implementation assumes that a DNS resolution failure is indicated by zookeeper_init() returning NULL and errno being set to EINVAL (Zk translates getaddrinfo() failures into errno values). However, the current Zk code does:

static int getaddrinfo_errno(int rc) {
    switch(rc) {
    case EAI_NONAME:
// ZOOKEEPER-1323 EAI_NODATA and EAI_ADDRFAMILY are deprecated in FreeBSD.
#if defined EAI_NODATA && EAI_NODATA != EAI_NONAME
    case EAI_NODATA:
#endif
        return ENOENT;
    case EAI_MEMORY:
        return ENOMEM;
    default:
        return EINVAL;
    }
}

getaddrinfo() returns EAI_NONAME when "the node or service is not known"; per discussion in MESOS-2186, this seems to happen intermittently due to DNS failures.

Proposed fix: looking at errno is always going to be somewhat fragile, but if we're going to continue doing that, we should check for ENOENT as well as EINVAL.

Issue Links

• is related to: MESOS-2186 Mesos crashes if any configured zookeeper does not resolve.
• is related to: MESOS-1326 Retry policy for zookeeper_init failures
• is related to: MESOS-1523 ZooKeeper timeout should be longer