Ok so I tried to make this work for an entire day and bottom line is that once I move the DocumentsWriter#abort() out of the sync block my test still fails all over the place. Yet, it's not hanging but concurrent access to IW while IW#deleteAll() is called is entirely broken IMO. I don't even know where to start, here is a small wrapup of the failures I saw:
- asserts are tripped in global field map since we clear and concurrently index (remember indexing is non-blocking)
- concurrent commits fail with fiel not found exception (even if we fully sync) seems like some state in IW is not cleared
- updatePendingMerges fails with FNF when merges are updated concurrently.
To begin with I doubt that the semantics of IW#deleteAll() are correct today if you are accessing the IW concurrently. I mean we basically dropping everything and don't maintain any happens before relationship here at all, delete all files that are not referenced in any seg info wipe all the global field infos etc. We should address this properly.
I agree that we have to fix this until 4.3.1!
Yet, Serguiuz do you see any FileNotFoundExceptions or anything when you concurrently call deleteAll? I mean this seems entirely broken to me at this point. I suggest you to use deleteQuery(new MatchAllDocsQuery()) for now and not lock globally.