OK I found the cause of the "expected:<1> but was:<0>" failures; it's
because SegmentInfos is failing to actually find the latest segments_N
The test commits frequently and reopens readers frequently w/ multiple
threads. Because our MockDirWrapper "acts like windows", we prevent
deleting segments_N files that are being read and so when a given
thread goes to reopen, the fallback logic in SegmentInfos can easily
open a too-old commit point.
To fix this I changed the retry order in which SegmentInfos tries to
find the segments file. Previously, upon failing to load segments_N
we would immediately attempt to load segments_(N-1). I've changed
that now to instead first try re-checking (re-list the dir and re-open
the segments.gen file), and only if re-checking produced the same "N"
do we then fallback to N-1.
I think this is actually an important change... back when this logic
was first created we did not have deletion policies and so a if the
segments_(N-1) existed it really did mean a commit to segments_N was
still in process. But today, w/ del policy, it could easily mean that
segments_(N-1) was just not deleted. So I think it's important to
re-list before trying segments_(N-1).
This fixes another failure case for this test, but, I still sometimes
see the "IndexFileDeleter does not know about XXX" (that other tests
have also hit), so I'll leave this open for now...