There's been quite a few jenkins failures from FullSolrCloudDistribCmdsTest that all seem to follow a similar pattern:
- Failure manifests as "Could not find collection:collection2"
- Failing seeds frequently reproduce, but aren't guaranteed to
- Root cause can be traced back to the collection creation failing because one of more replica cores failed due to the brand new (Solr)IndexWriter expects to find an existing segments file
- SolrCore should have already created an (empty) index in SolrCore.initIndex(...)
- The fact that the SolrIndexWrite throws this exception in it's constructor suggests that the earlier call to SolrCore.initIndex(...) is not functioning reliably
- Based on some experimenting i've done, it seems like the underlying problem is that in SolrCore.initIndex(...) the DirectoryFactory can "lie" about wether a directory already exists.
More details to follow in comments.