[#LUCENE-1044] Behavior on hard power shutdown

Lucene - Java

Behavior on hard power shutdown

Created: 02/Nov/07 07:07 PM Updated: 23/Feb/09 05:15 PM

Return to search

Component/s:

Index

Affects Version/s:

None

Fix Version/s:

2.4

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
FSyncPerfTest.java	2007-11-22 02:16 PM	Doron Cohen	6 kB
LUCENE-1044.patch	2007-11-04 03:28 PM	Michael McCandless	6 kB
LUCENE-1044.take2.patch	2007-11-04 06:29 PM	Michael McCandless	7 kB
LUCENE-1044.take3.patch	2007-11-06 10:38 PM	Michael McCandless	16 kB
LUCENE-1044.take4.patch	2007-11-27 08:14 PM	Michael McCandless	7 kB
LUCENE-1044.take5.patch	2007-12-11 08:20 PM	Michael McCandless	86 kB
LUCENE-1044.take6.patch	2008-02-05 10:52 AM	Michael McCandless	194 kB
LUCENE-1044.take7.patch	2008-02-05 09:19 PM	Michael McCandless	205 kB
LUCENE-1044.take8.patch	2008-02-08 08:31 PM	Michael McCandless	206 kB

Environment:

Windows Server 2003, Standard Edition, Sun Hotspot Java 1.5

Lucene Fields:	New
Resolution Date:	11/Feb/08 06:59 PM

Description

« Hide

When indexing a large number of documents, upon a hard power failure (e.g. pull the power cord), the index seems to get corrupted. We start a Java application as an Windows Service, and feed it documents. In some cases (after an index size of 1.7GB, with 30-40 index segment .cfs files) , the following is observed.

The 'segments' file contains only zeros. Its size is 265 bytes - all bytes are zeros.
The 'deleted' file also contains only zeros. Its size is 85 bytes - all bytes are zeros.

Before corruption, the segments file and deleted file appear to be correct. After this corruption, the index is corrupted and lost.

This is a problem observed in Lucene 1.4.3. We are not able to upgrade our customer deployments to 1.9 or later version, but would be happy to back-port a patch, if the patch is small enough and if this problem is already solved.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#593799	Sat Nov 10 17:51:00 UTC 2007	mikemccand	~~LUCENE-1044~~: add doSync option to FSDirectory.getDirectory, defaulting to true, to sync() each file descriptor before close()
				Files Changed
				MODIFY /lucene/java/trunk/src/test/org/apache/lucene/store/TestLockFactory.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestIndexModifier.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestNorms.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestCompoundFile.java MODIFY /lucene/java/trunk/CHANGES.txt MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestThreadedOptimize.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/store/FSDirectory.java MODIFY /lucene/java/trunk/contrib/benchmark/src/java/org/apache/lucene/benchmark/byTask/package.html MODIFY /lucene/java/trunk/contrib/benchmark/src/java/org/apache/lucene/benchmark/byTask/PerfRunData.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestStressIndexing.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestAtomicUpdate.java

Repository	Revision	Date	User	Message
ASF	#594824	Wed Nov 14 10:25:34 UTC 2007	mikemccand	~~LUCENE-1044~~: default doSync to false for now because performance cost of the simple 'sync() before every close()' approach is too severe in many cases
				Files Changed
				MODIFY /lucene/java/trunk/src/java/org/apache/lucene/store/FSDirectory.java

Repository	Revision	Date	User	Message
ASF	#596874	Tue Nov 20 23:17:44 UTC 2007	mikemccand	~~LUCENE-1044~~: also re-default doSync back to false in contrib/benchmark
				Files Changed
				MODIFY /lucene/java/trunk/contrib/benchmark/src/java/org/apache/lucene/benchmark/byTask/PerfRunData.java

Repository	Revision	Date	User	Message
ASF	#597371	Thu Nov 22 10:46:18 UTC 2007	mikemccand	~~LUCENE-1044~~: put back original ctor for FSIndexOutput
				Files Changed
				MODIFY /lucene/java/trunk/src/java/org/apache/lucene/store/FSDirectory.java

Repository	Revision	Date	User	Message
ASF	#602165	Fri Dec 07 17:42:33 UTC 2007	mikemccand	~~LUCENE-1044~~: revert the doSync option to FSDirectory
				Files Changed
				MODIFY /lucene/java/trunk/src/test/org/apache/lucene/store/TestLockFactory.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestIndexModifier.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestNorms.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestCompoundFile.java MODIFY /lucene/java/trunk/CHANGES.txt MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestThreadedOptimize.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/store/FSDirectory.java MODIFY /lucene/java/trunk/contrib/benchmark/src/java/org/apache/lucene/benchmark/byTask/package.html MODIFY /lucene/java/trunk/contrib/benchmark/src/java/org/apache/lucene/benchmark/byTask/PerfRunData.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestStressIndexing.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestAtomicUpdate.java

Repository	Revision	Date	User	Message
ASF	#620576	Mon Feb 11 18:56:09 UTC 2008	mikemccand	~~LUCENE-1044~~: sync index files in IndexWriter to ensure index is intact if machine or OS crashes
				Files Changed
				MODIFY /lucene/java/trunk/src/java/org/apache/lucene/store/Directory.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestDeletionPolicy.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java MODIFY /lucene/java/trunk/docs/fileformats.html MODIFY /lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestMultiSegmentReader.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/store/MockRAMOutputStream.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestIndexReader.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestIndexFileDeleter.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/store/MockRAMDirectory.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/IndexFileDeleter.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/store/FSDirectory.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/store/MockRAMInputStream.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestIndexWriterDelete.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestAtomicUpdate.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/DocumentsWriter.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/SegmentInfos.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestIndexModifier.java ADD /lucene/java/trunk/src/test/org/apache/lucene/index/TestCrash.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/IndexWriter.java ADD /lucene/java/trunk/src/java/org/apache/lucene/store/ChecksumIndexInput.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestIndexWriter.java MODIFY /lucene/java/trunk/docs/fileformats.pdf MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/CheckIndex.java MODIFY /lucene/java/trunk/CHANGES.txt MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/DirectoryIndexReader.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestThreadedOptimize.java ADD /lucene/java/trunk/src/java/org/apache/lucene/store/ChecksumIndexOutput.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestStressIndexing.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/util/LuceneTestCase.java

Repository	Revision	Date	User	Message
ASF	#627315	Wed Feb 13 09:04:34 UTC 2008	mikemccand	~~LUCENE-1044~~: fix CheckIndex tool to properly identify 2.4 index format
				Files Changed
				MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/CheckIndex.java