[SOLR-14504] ZkController LiveNodesListener has NullPointerException in startup race

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 7.7, 7.7.1, 7.7.2, 7.7.3, 8.0, 8.1, 8.1.1, 8.2, 8.3, 8.3.1, 8.4, 8.4.1, 8.5, 8.5.1
• Fix Version/s: 8.6
• Assignee: Andrzej Bialecki
• Reporter: Colvin Cowie
• Created: 21/May/20 11:37
• Updated: 08/Aug/23 14:52
• Resolved: 04/May/22 21:45

Description

If a NODELOST event happens before the cloudManager is initialized then a NullPointerException will occur on this line https://github.com/apache/lucene-solr/blob/c18666ad05afc02979c150aacd4810cff02e43f3/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L1020

byte[] json = Utils.toJSON(Collections.singletonMap("timestamp", cloudManager.getTimeSource().getEpochTimeNs())); 

Rather than accessing cloudManager directly, getSolrCloudManager() should be called.

 

This happens very rarely, but if it happens it will stop Solr starting, result in "CoreContainer is either not initialized or shutting down". Snippet from 8.3.1

2020-05-19 03:44:40.241 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2020-05-19 03:44:40.245 INFO  (zkConnectionManagerCallback-11-thread-1) [   ] o.a.s.c.c.ConnectionManager zkClient has connected
2020-05-19 03:44:40.245 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2020-05-19 03:44:40.359 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
2020-05-19 03:44:40.361 INFO  (zkConnectionManagerCallback-13-thread-1) [   ] o.a.s.c.c.ConnectionManager zkClient has connected
2020-05-19 03:44:40.361 INFO  (main) [   ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
2020-05-19 03:44:40.417 INFO  (main) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
2020-05-19
 03:44:56.606 INFO  (zkCallback-12-thread-2) [   ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> 
(0)
2020-05-19 03:44:56.614 ERROR (main) [   ] 
o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home 
property and the logs
2020-05-19 03:44:56.639 ERROR (main) [   ] o.a.s.c.SolrCore null:java.lang.NullPointerException
	at org.apache.solr.cloud.ZkController.lambda$registerLiveNodesListener$10(ZkController.java:1020)
	at org.apache.solr.common.cloud.ZkStateReader.registerLiveNodesListener(ZkStateReader.java:880)
	at org.apache.solr.cloud.ZkController.registerLiveNodesListener(ZkController.java:1035)
	at org.apache.solr.cloud.ZkController.init(ZkController.java:917)
	at org.apache.solr.cloud.ZkController.<init>(ZkController.java:473)
	at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:115)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:631)

 

 

Issue Links

• is caused by: SOLR-13072 Management of markers for nodeLost / nodeAdded events is broken
• is duplicated by: SOLR-14505 Intermittent NPE in ZkController.registerLiveNodesListener()