RecoveryAfterSoftCommitTest is failing frequently on Jenkins because the test only gives 30 seconds for the replica to recover after healing the partition. It looks like it's taking >30 seconds to replicate the full index from the leader (the test is designed so that peer sync can't work). It seems bad that it takes >30 seconds to replicate an index with only 115 documents in it ... wonder if there is cruft laying around from other tests? I've run beast on this test locally and it always passes. What's weird is I see log messages like:
[junit4] 2> 1436627 T4242 N:127.0.0.1:63274_ecb%2Fay C476 oash.IndexFetcher.fetchLatestIndex Number of files in latest index in master: 263
263 files for an index with 115 docs? Doesn't seem right!