Sorry for the delayed response but i was trying to collect some data, I still have to wait for the log created by lucene APIs after turning on infostream, i will post that log file as soon as i get it. To answer your questions, i am running on Windows 2003 server with IBM JRE 1.6 and I am trying to index a database that is about 45 GB in size. On Unix, i am not facing this issue.
With the lucene logging on, i waited for about 2 days for the NPE to reoccur, but it did not happen. I was tyring with the stock jar file of lucene 2.3.2. However, indexing failed again and the index file was again deleted. Looking into the logs this time, I found the following exceptions:
16:06:48 03/01/2010: Fatal error while indexing CRs.
Stack trace: java.io.FileNotFoundException: index_21f.fdt (The system cannot find the file specified.)
at java.io.RandomAccessFile.open(Native Method)
I am not sure if deletion of the index file when these exceptions are thrown, is the right behaviour.
I am re-running my test to try and reproduce the NPE issue which i had reported earlier and i intend to pass on the logs to you as soon as i have it. However, are you aware of any issue that a combination of IBM JRE + Windows 2003 server would have with lucene?
As far as the code is concern I am closing the indexwriter in finally block, when is thrown I am logging it in catch block and then finally closing the indexwriter.