Since upgrading to 7.2.1, we ran into an issue where Solr ignores aliases.json file stored in ZooKeeper.
- SolrCloud cluster is down
- Direct update of aliases.json file in ZooKeeper with Solr ZkCLI without using Collections API :
- java ... org.apache.solr.cloud.ZkCLI -zkhost ... -cmd clear /aliases.json
- java ... org.apache.solr.cloud.ZkCLI -zkhost ... -cmd put /aliases.json "new content"
- SolrCloud cluster is started => aliases.json not taken into account
Digging a bit in the code, what is actually causing the issue is that, when starting, Solr now checks for the metadata of the aliases.json file and if the version metadata from ZooKeeper is lower or equal to local version, it keeps the local version.
When it starts, Solr has a local version of 0 for the aliases but ZooKeeper also has a version of 0 of the file because we just recreated it. So Solr ignores ZooKeeper configuration and never has a chance to load aliases.
Relevant parts of Solr code are:
- https://github.com/apache/lucene-solr/blob/branch_7_2/solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java : line 1562 : method setIfNewer
- https://github.com/apache/lucene-solr/blob/branch_7_2/solr/solrj/src/java/org/apache/solr/common/cloud/Aliases.java : line 45 : the "empty" Aliases object with default version 0
Note that a workaround is to force ZooKeeper to always have a version greater than 0 for aliases.json file (for instance by not clearing the file and just overwriting it again and again).