Major Description
When using secure zookeeper and a valid jaas configuration, SolrCloud will reject updates with a 503 error.
The problem is that in this case ZooKeeper sends states to the watchers like "SaslAuthenticated", but the ConnectionManager treats any state it doesn't know about as a disconnect. Then, whenever a request comes in, SolrCloud will think it can't talk to ZooKeeper and reject the request.
These are the valid states that watchers can see with the current ZooKeeper version (3.4.5):
Disconnected
SyncConnected
AuthFailed
ConnectedReadOnly
SaslAuthenticated
Expired
ConnectionManager currently does not handle:
SaslAuthenticated
ConnectedReadOnly
AuthFailed
From my tests, it seems like the correct thing to do is just ignore these states:
1) SaslAuthenticated - nothing has gone wrong here, so no need to disconnected
2) ConnectedReadOnly - the client would only see this state if they specifically specified a read only client, so this is expected
3) AuthFailed - this one is a little tricky. If you try to authentication but fail (say you have an invalid JAAS conf), you may get AuthFailed, but depending on the ZK settings, you may still be able to communicate with ZooKeeper (i.e. see this in the log: Will continue connection to Zookeeper server \
without SASL authentication, if Zookeeper server allows it.)
So, it seems correct to ignore these states, possibly warning on AuthFailed, because that isn't expected in proper operation.