"Cannot rename segments.new to segments"

• Key: LUCENE-425
• Type: Bug
• Status: Closed
• Resolution: Duplicate
• Priority: Major
• Assignee: Unassigned
• Reporter: Mikko Noromaa
• Component/s: Index
• Affects Version/s: 1.4

Environment

Operating System: Windows XP
Platform: PC

Description

AddDocument periodically fails with "Cannot rename segments.new to segments".
Sometimes it happens after 10,000 documents, sometimes after 200,000 documents.

My index is placed on a FAT partition. My Lucene version is 1.4.3. I am using
a single IndexWriter object and doing all AddDocuments calls serially from a
single thread.

I saw some discussion about this problem in the mailing lists, but looks like
the problem wasn't completely eliminated yet.

My suggested solutions are as follows:

• If renameTo() fails, wait some time and retry. After this, apply the other
  methods that are now in FSDirectory.java. Retry them as well after waiting for
  some time.

• In the IndexWriter constructor, check for 'segments.new' if 'segments'
  cannot be found. This would allow recovery if FSDirectory failed on a previous
  run. Now I have to write the recovery code myself outside of Lucene.

Comments

[Michael McCandless]

There was a recent thread on java-user that looks very similar to this issue:

http://www.gossamer-threads.com/lists/lucene/java-user/38669

When Lucene writes a new "segments" or new "deletable" file it first
writes to X.new and then renames X.new --> X. Apparently that
renameTo can sometimes fail, even though both the X.new and the X file
are "closed" from Java.

The "best guess" from that discussion is that there is a JVM bug on
certain JVMs on Windows, whereby a file handle (at the Windows OS
level) is held open even after from Java it has been "closed".

Ant has a workaround for this issue: force a GC, wait, and then retry.
I don't really like that workaround for Lucene because we should not
suddenly / unexpectedly do a system level action like forcing GC –
it's a "leaky abstraction"
(http://www.joelonsoftware.com/articles/LeakyAbstractions.html).

Which JVM are you using? I couldn't reproduce this; I made a small
change to the IndexFiles demo command-line tool and then refreshed the
view from Windows Explorer. On Sun JDK 1.4.2_10 and Sun JDK 1.5.0_07
I couldn't get the IOException.

Issue LUCNE-554 also proposes more robust logic around opening the
.NEW file if it exists. I think that makes alot of sense.

[Michael McCandless]

I think this one is a dup of LUCENE-665

The intermittant errors on WIN32 came down to 3rd party software that can open files shortly after they were created, thus preventing rename from succeeding. The combination of native LockFactor implementation (LUCENE-678 ) and Lockless commits (LUCENE-701 ) should resolve these issues.

We have been using Lucene 1.9.1 for the past 2 years and this issue occurs on a regular basis. Always on Windows and more often on machines with slower hard disks, i.e. Laptops. A colleague of mine downloaded the source and patched it to retry the rename and delete operations several times before giving up. This works well on our environments so we would like to get the change into Trunk code.

I will prepare a patch against trunk shortly which will solve the problem.

[Michael McCandless]

I believe the root cause of this issue was resolved with LUCENE-701: we no longer rename files when writing a Lucene index.

Are you able to get this issue to happen with Lucene 2.2?