[KUDU-1099] Consensus should be more resilient to transient DNS failures

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Private Beta
• Component/s: consensus
• Target Version/s: Public beta
• Code Review: http://gerrit.sjc.cloudera.com:8080/7870
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 02/Sep/15 12:58
• Updated: 16/Feb/18 00:36
• Resolved: 08/Sep/15 20:48

Description

When starting up the Kudu cluster on bolt80 I often see a couple nodes crash with errors like:

F0902 10:12:03.816699 37993 leader_election.cc:157] Check failed: _s.ok() Bad status: Network error: Unable to resolve address 'e1313.halxg.cloudera.com': Name or service not known

I'm guessing that we end up producing a DNS storm of some kind, and this somehow causes us to get some incorrect "host not found" errors. We shouldn't crash the whole process.

Issue Links

• is related to: KUDU-1839 DNS failure during tablet creation lead to undeletable tablet
• is related to: KUDU-2302 Leader crashes if it can't resolve DNS address of a peer