With 'DynamicReconfig' feature in v3.5.6, ideally the servers can be added and removed without restarting ZooKeeper service on any of the nodes.
But, with Keberos (GSSAPI via SASL) enabled quorum authentication/authorization, this is not possible. Because, when you try to add a new server, it won't be able to connect to any of the members in the ensemble and the data won't be synced. This is because all the members reject it based on authorization. For this to make it work, we need to do 'reconfig', then restart leader, the new member and rest of the members.
Is this the expected behavior with Quorum-auth + DynamicReconfig? Or am I missing something here.
This is our basic quorum-auth config:
FTR: I raised this question in ZooKeeper-user forum and both Mate and Enrico suspect this to be a bug.
Also this is easily reproducible in a Kerbers (GSSAPI via SASL) enabled quorum based ensemble.