[SOLR-8697] Fix LeaderElector issues

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.4.1
• Fix Version/s: 5.5.1, 6.0
• Component/s: SolrCloud
• Labels: patch reliability solrcloud
• Assignee: Mark Miller
• Reporter: Scott Blum
• Created: 18/Feb/16 19:54
• Updated: 02/Oct/19 17:25
• Resolved: 21/Apr/16 21:12

Description

This patch is still somewhat WIP for a couple of reasons:

1) Still debugging test failures.
2) This will more scrutiny from knowledgable folks!

There are some subtle bugs with the current implementation of LeaderElector, best demonstrated by the following test:

1) Start up a small single-node solrcloud. it should be become Overseer.
2) kill -9 the solrcloud process and immediately start a new one.
3) The new process won't become overseer. The old process's ZK leader elect node has not yet disappeared, and the new process fails to set appropriate watches.

NOTE: this is only reproducible if the new node is able to start up and join the election quickly.

Issue Links

• is duplicated by: SOLR-6498 LeaderElector sometimes will appear multiple ephemeral nodes in the zookeeper
• is related to: SOLR-8837 Duplicate leader elector node detection is broken