[#LUCENE-756] Maintain norms in a single file .nrm

Lucene - Java

Maintain norms in a single file .nrm

Created: 21/Dec/06 02:10 AM Updated: 27/Feb/07 06:10 PM

Return to search

Component/s:

Index

Affects Version/s:

None

Fix Version/s:

2.1

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
index.premergednorms.cfs.zip	2007-01-16 04:36 PM	Michael McCandless	2 kB
index.premergednorms.nocfs.zip	2007-01-16 04:36 PM	Michael McCandless	5 kB
LUCENE-756-Jan16.patch	2007-01-16 04:36 PM	Michael McCandless	16 kB
LUCENE-756-Jan16.Take2.patch	2007-01-16 05:50 PM	Michael McCandless	18 kB
nrm.patch.2.txt	2006-12-22 08:51 AM	Doron Cohen	16 kB
nrm.patch.3.txt	2007-01-03 10:49 PM	Doron Cohen	23 kB
nrm.patch.txt	2006-12-21 04:26 AM	Doron Cohen	12 kB

Lucene Fields:	Patch Available
Resolution Date:	16/Jan/07 08:28 PM

Description

« Hide

Non-compound indexes are ~10% faster at indexing, and perform 50% IO activity comparing to compound indexes. But their file descriptors foot print is much higher.

By maintaining all field norms in a single .nrm file, we can bound the number of files used by non compound indexes, and possibly allow more applications to use this format.

More details on the motivation for this in: http://www.nabble.com/potential-indexing-perormance-improvement-for-compound-index---cut-IO---have-more-files-though-tf2826909.html (in particular http://www.nabble.com/Re%3A-potential-indexing-perormance-improvement-for-compound-index---cut-IO---have-more-files-though-p7910403.html).

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#493641	Sun Jan 07 04:19:21 UTC 2007	yonik	Maintain norms in a single file .nrm: ~~LUCENE-756~~
				Files Changed
				MODIFY /lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/SegmentReader.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/IndexWriter.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/IndexFileNames.java MODIFY /lucene/java/trunk/CHANGES.txt MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/SegmentInfo.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/SegmentMerger.java

Repository	Revision	Date	User	Message
ASF	#493897	Mon Jan 08 01:05:19 UTC 2007	yonik	norms test: ~~LUCENE-756~~
				Files Changed
				ADD /lucene/java/trunk/src/test/org/apache/lucene/index/TestNorms.java

Repository	Revision	Date	User	Message
ASF	#496851	Tue Jan 16 20:24:52 UTC 2007	mikemccand	~~LUCENE-756~~: small improvement to not rely on IO operation (fileExists) to check whether a "single norm file" is in use for the segment. Instead, save this information per segment explicitly into the segment infos file. Also renamed to "singleNormFile".
				Files Changed
				MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestSegmentMerger.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/SegmentInfos.java MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java MODIFY /lucene/java/trunk/src/site/src/documentation/content/xdocs/fileformats.xml MODIFY /lucene/java/trunk/src/test/org/apache/lucene/index/TestDoc.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/SegmentReader.java MODIFY /lucene/java/trunk/src/java/org/apache/lucene/index/SegmentInfo.java