[SOLR-13376] Multi-node race condition to create/remove nodeLost markers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.3
• Assignee: Andrzej Bialecki
• Reporter: Chris M. Hostetter
• Created: 04/Apr/19 21:19
• Updated: 08/Oct/19 12:31
• Resolved: 08/Oct/19 12:31

Description

NodeMarkersRegistrationTest.testNodeMarkersRegistration is frequently failing on jenkins builds in the same spot, with a similar looking logs.

Although i haven't been able to reproduce these failures locally, I am fairly confident that the problem is a race condition bug that exists between when/how a new Overseer will process & clean up "nodeLost" marker's in ZK, with how other nodes may (mistakenly) re-create those markers in their liveNodes listener.