you are right, it is very weird that 1) this problem happens and 2) my patch fixes it. It took me quite a while to figure out what the real problem is but I think I have at least a guess now.
The problem again only happens on an IBM JVM. It happens rarely with your unmodified TestStressIndexing. If I change this testcase so that is does not perform the stress test on a RAMDirectory but only on a FSDirectory it happens almost always. This is VERY weird, because I did not change the stress test at all... I just commented the lines
// // First in a RAM directory:
// Directory directory = new RAMDirectory();
and suddenly every test fails with the IOException "The handle is invalid". So that makes me think it has something to do with garbage collection and the finalize() methods.
Now if I just comment out the finalize() method of FSDirectory.FSIndexOutput the test passes. This is even stranger, because the IOException occurred in FSDirectory.FSIndexInput.close(), not in FSIndexOutput, so in a different class which uses hence a different file descriptor.
So I checked how java.io.FileDescriptor is implemented: It simply wraps an int value. I can't see how these int values are computed, because the open() method of RandomAccessFile is native.
So I believe the following happens:
(1) FSIndexOutput uses a FileDescriptor with value x
(2) FSIndexOutput.close() is called, so the underlying file is being closed
(3) A new FSIndexInput instance is created by a searcher thread. This opens a RandomAccessFile. Because FileDescriptor x is not in use anymore, x is used as the value for the new FileDescriptor.
(4) Now garbage collection kicks in. It removes the old instance of FSIndexOutput for which close() has been called already. So the garbage collector calls finalize() which calls RandomAccessFile.close() again which still uses the descriptor with value x. So this call of close() actually closes the file used by the IndexInput instance created in (3).
(5) FSIndexInput.close() is called and tries to close the file which has been closed already in (4) and thus the IOException occurs.
So it seems to me that the IBM JVM makes file descriptor values available after a file has been closed, whereas Sun waits until the FileDescriptor instance is destroyed. This might be a bug in the JVM, but since this patch is very simple we could just use it to be on the safe side.
Do you think this makes sense? Or does anybody have a better idea why commenting out the finalize() method in FSIndexOutput prevents FSIndexInput.close() from throwing the IOException?