MMapDirectory can't create new index on Windows

• Key: LUCENE-727
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Michael McCandless
• Reporter: Benson Margulies
• Component/s: Index
• Affects Version/s: 2.0.0

Description

When I set the system property to request the use of the mmap directory, and start building a large index, the process dies with an IOException trying to delete a file. Apparently, Lucene isn't closing down the memory map before deleting the file.

Comments

[Hoss Man]

Can you inlcude a full stack trace of the exception so we can see what code path causes this problem?

it would be even better if you could inlcude a JUnit test demonstrating the problem.

[Benson Margulies]

Here you go.

[Hoss Man]

Per robert engels on java-dev...

> This is a known JDK issue with memory mapping a file.
>
> The references may not be GC'd at the time the request to delete is
> performed, and thus you will get an IOException.
>
> See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4715154

...so it may not be possible to fix this.

[Michael McCandless]

Thank you Benson for the unit test and opening this issue!

Indeed, when I run your test, I see the exception in Lucene-2.0.0, but, not on the trunk.

It looks like lockless commits (LUCENE-701 ) has fixed this because the files in the index are now "write once". Specifically, we no longer try to rename any files (nor do we even use the "deletable" file anymore). The general philosopy is to rely on as little from the filesystem as we possibly can, so as to maximize portability...

This is the exception on Lucene 2.0.0:

[junit] Testcase: testMmapIndex(org.apache.lucene.index.TestWindowsMMap): Caused an ERROR
[junit] Cannot delete F:\mike\src\lucene-2.0.0\testLuceneMmap\deletable
[junit] java.io.IOException: Cannot delete F:\mike\src\lucene-2.0.0\testLuceneMmap\deletable
[junit] at org.apache.lucene.store.FSDirectory.renameFile(FSDirectory.java:268)
[junit] at org.apache.lucene.index.IndexWriter.writeDeleteableFiles(IndexWriter.java:827)
[junit] at org.apache.lucene.index.IndexWriter.deleteFiles(IndexWriter.java:778)
[junit] at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:747)
[junit] at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:686)
[junit] at org.apache.lucene.index.IndexWriter.maybeMergeSegments(IndexWriter.java:674)
[junit] at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:479)
[junit] at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:462)
[junit] at org.apache.lucene.index.TestWindowsMMap.testMmapIndex(TestWindowsMMap.java:61)

Since this is fixed on trunk, I'll resolve it as fixed.

But first I'll commit the new unit test (thanks Benson)!