This defaults to the current value of {@link Similarity#getDefault()}.
+ */
+ public Similarity getSimilarity();
+
+ /**
+ * Expert: Set the interval between indexed terms. Large values cause less
+ * memory to be used by IndexReader, but slow random-access to terms. Small
+ * values cause more memory to be used by an IndexReader, and speed
+ * random-access to terms.
+ *
+ * This parameter determines the amount of computation required per query
+ * term, regardless of the number of documents that contain that term. In
+ * particular, it is the maximum number of other terms that must be
+ * scanned before a term is located and its frequency and position information
+ * may be processed. In a large index with user-entered query terms, query
+ * processing time is likely to be dominated not by term lookup but rather
+ * by the processing of frequency and positional data. In a small index
+ * or when many uncommon query terms are generated (e.g., by wildcard
+ * queries) term lookup may become a dominant cost.
+ *
+ * In particular, numUniqueTerms/interval terms are read into
+ * memory by an IndexReader, and, on average, interval/2 terms
+ * must be scanned for each random term access.
+ *
+ * @see #DEFAULT_TERM_INDEX_INTERVAL
+ */
+ public void setTermIndexInterval(int interval);
+
+ /**
+ * Expert: Return the interval between indexed terms.
+ *
+ * @see #setTermIndexInterval(int)
+ */
+ public int getTermIndexInterval();
+
+ /**
+ * Determines the minimal number of documents required before the buffered
+ * in-memory documents are merging and a new Segment is created.
+ * Since Documents are merged in a {@link org.apache.lucene.store.RAMDirectory},
+ * large value gives faster indexing. At the same time, mergeFactor limits
+ * the number of files open in a FSDirectory.
+ *
+ *
The default value is 10. + * + * + * @throws IllegalArgumentException if maxBufferedDocs is smaller than 2 + */ + public void setMaxBufferedDocs(int maxBufferedDocs); + + /** + * + * + * @see #setMaxBufferedDocs + */ + public int getMaxBufferedDocs(); + + /** + * The maximum number of terms that will be indexed for a single field in a + * document. This limits the amount of memory required for indexing, so that + * collections with very large files will not crash the indexing process by + * running out of memory.
+ * Note that this effectively truncates large documents, excluding from the + * index terms that occur further in the document. If you know your source + * documents are large, be sure to set this value high enough to accomodate + * the expected size. If you set it to Integer.MAX_VALUE, then the only limit + * is your memory, but you should anticipate an OutOfMemoryError. + * By default, no more than 10,000 terms will be indexed for a field. + */ + public void setMaxFieldLength(int maxFieldLength); + + /** + * + * + * @see #setMaxFieldLength + */ + public int getMaxFieldLength(); + + /** + * Determines the largest number of documents ever merged by addDocument(). + * Small values (e.g., less than 10,000) are best for interactive indexing, + * as this limits the length of pauses while indexing to a few seconds. + * Larger values are best for batched indexing and speedier searches. + * + *The default value is {@link Integer#MAX_VALUE}. + */ + public void setMaxMergeDocs(int maxMergeDocs); + + /** + * + * + * @see #setMaxMergeDocs + */ + public int getMaxMergeDocs(); + + /** + * Determines how often segment indices are merged by addDocument(). With + * smaller values, less RAM is used while indexing, and searches on + * unoptimized indices are faster, but indexing speed is slower. With larger + * values, more RAM is used during indexing, and while searches on unoptimized + * indices are slower, indexing is faster. Thus larger values (> 10) are best + * for batch index creation, and smaller values (< 10) for indices that are + * interactively maintained. + * + *
This must never be less than 2. The default value is 10. + */ + public void setMergeFactor(int mergeFactor); + + /** + * + * + * @see #setMergeFactor + */ + public int getMergeFactor(); + + /** + * Sets the maximum time to wait for a write lock (in milliseconds). + */ + public void setWriteLockTimeout(long writeLockTimeout); + + /** + * + * + * @see #setWriteLockTimeout + */ + public long getWriteLockTimeout(); + + /** + * Sets the maximum time to wait for a commit lock (in milliseconds). + */ + public void setCommitLockTimeout(long commitLockTimeout); + + /** + * + * + * @see #setCommitLockTimeout + */ + public long getCommitLockTimeout(); + + /** Expert: Set the TokenSelector used to determine subset of tokens stored in term vectors. + * @param selector the term vector TokenSelector + */ + public void setTermVectorTokenSelector(TokenSelector selector); + + /** Expert: Set the TokenSelector used to determine subset of tokens stored in term vectors. + * @return the TokenSelector used to determine term vector tokens + */ + public TokenSelector getTermVectorTokenSelector(); + + /** If non-null, information about merges and a message when + * maxFieldLength is reached will be printed to this. + */ + public void setInfoStream(PrintStream infoStream); + + /** + * + * + * @see #setInfoStream + */ + public PrintStream getInfoStream(); + +} Index: src/java/org/apache/lucene/index/IndexWriter.java =================================================================== --- src/java/org/apache/lucene/index/IndexWriter.java (revision 414705) +++ src/java/org/apache/lucene/index/IndexWriter.java (working copy) @@ -17,6 +17,7 @@ */ import org.apache.lucene.analysis.Analyzer; +import org.apache.lucene.analysis.TokenSelector; import org.apache.lucene.document.Document; import org.apache.lucene.search.Similarity; import org.apache.lucene.store.Directory; @@ -56,7 +57,7 @@ @see IndexModifier IndexModifier supports the important methods of IndexWriter plus deletion */ -public class IndexWriter { +public class IndexWriter implements Writable { /** * Default value for the write lock timeout (1,000). @@ -100,8 +101,10 @@ */ public final static int DEFAULT_TERM_INDEX_INTERVAL = 128; - private Directory directory; // where this index resides - private Analyzer analyzer; // how to analyze text + private Directory directory; // where this index resides + private Analyzer analyzer; // how to analyze text + private TokenSelector termVectorTokenSelector; // subset of token stream stored in term vectors + private TokenSelector positionsTokenSelector; // subset of token stream for which positions are stored private Similarity similarity = Similarity.getDefault(); // how to normalize @@ -153,6 +156,38 @@ return this.similarity; } + /** Expert: Set the TokenSelector used to determine subset of tokens stored in term vectors. + * @param selector the term vector TokenSelector + */ + public void setTermVectorTokenSelector(TokenSelector selector) { + this.termVectorTokenSelector = selector; + } + + /** Expert: Set the TokenSelector used to determine subset of tokens stored in term vectors. + * @return the TokenSelector used to determine term vector tokens + */ + public TokenSelector getTermVectorTokenSelector() { + return termVectorTokenSelector; + } + + /** Expert: Set the TokenSelector used to determine subset of tokens for which positions are stored. + * (At least one position is always stored for each term in each doc to ensure the term stays in + * the index so long as any docs reference it) + * @param selector the positions TokenSelector + */ + public void setPositionsTokenSelector(TokenSelector selector) { + this.positionsTokenSelector = selector; + } + + /** Expert: Set the TokenSelector used to determine subset of tokens for which freq and positions are stored.. + * (At least one position is always stored for each term in each doc to ensure the term stays in + * the index so long as any docs reference it) + * @return the positions TokenSelector + */ + public TokenSelector getPositionsTokenSelector() { + return positionsTokenSelector; + } + /** Expert: Set the interval between indexed terms. Large values cause less * memory to be used by IndexReader, but slow random-access to terms. Small * values cause more memory to be used by an IndexReader, and speed @@ -471,6 +506,8 @@ public void addDocument(Document doc, Analyzer analyzer) throws IOException { DocumentWriter dw = new DocumentWriter(ramDirectory, analyzer, this); + dw.setTermVectorTokenSelector(termVectorTokenSelector); + dw.setPositionsTokenSelector(positionsTokenSelector); dw.setInfoStream(infoStream); String segmentName = newSegmentName(); dw.addDocument(segmentName, doc); Index: src/java/org/apache/lucene/index/Writable.java =================================================================== --- src/java/org/apache/lucene/index/Writable.java (revision 0) +++ src/java/org/apache/lucene/index/Writable.java (revision 0) @@ -0,0 +1,262 @@ +/* + * Writable.java + * + * Created on April 28, 2006, 6:10 PM + * + */ + +package org.apache.lucene.index; + +import java.io.IOException; +import java.io.PrintStream; +import org.apache.lucene.analysis.Analyzer; +import org.apache.lucene.analysis.TokenSelector; +import org.apache.lucene.document.Document; +import org.apache.lucene.search.Similarity; +import org.apache.lucene.store.Directory; + +/** + * An interface that abstracts index writers (e.g., IndexWriter, ParallelWriter) + * + * @author Chuck Williams + */ +public interface Writable { + + /** + * Adds a document to this index. If the document contains more than + * {@link #setMaxFieldLength(int)} terms for a given field, the remainder are + * discarded. + */ + public void addDocument(Document doc) throws IOException; + + /** + * Adds a document to this index, using the provided analyzer instead of the + * value of {@link #getAnalyzer()}. If the document contains more than + * {@link #setMaxFieldLength(int)} terms for a given field, the remainder are + * discarded. + */ + public void addDocument(Document doc, Analyzer analyzer) throws IOException; + + /** + * Returns the number of documents currently in this index. + */ + public int docCount(); + + /** + * Merges all segments together into a single segment, optimizing an index + * for search. + */ + public void optimize() throws IOException; + + /** + * Flushes all changes to an index and closes all associated files. + */ + public void close() throws IOException; + + /** + * Returns the analyzer used by this index. + */ + public Analyzer getAnalyzer(); + + + /** + * Setting to turn on usage of a compound file. When on, multiple files + * for each segment are merged into a single file once the segment creation + * is finished. This is done regardless of what directory is in use. + */ + public void setUseCompoundFile(boolean value); + + /** + * Get the current setting of whether to use the compound file format. + * Note that this just returns the value you set with setUseCompoundFile(boolean) + * or the default. You cannot use this to query the status of an existing index. + * + * @see #setUseCompoundFile(boolean) + */ + public boolean getUseCompoundFile(); + + /** + * Expert: Set the Similarity implementation used by this IndexWriter. + * + * @see Similarity#setDefault(Similarity) + */ + public void setSimilarity(Similarity similarity); + + /** + * Expert: Return the Similarity implementation used by this IndexWriter. + * + *
This defaults to the current value of {@link Similarity#getDefault()}.
+ */
+ public Similarity getSimilarity();
+
+ /**
+ * Expert: Set the interval between indexed terms. Large values cause less
+ * memory to be used by IndexReader, but slow random-access to terms. Small
+ * values cause more memory to be used by an IndexReader, and speed
+ * random-access to terms.
+ *
+ * This parameter determines the amount of computation required per query
+ * term, regardless of the number of documents that contain that term. In
+ * particular, it is the maximum number of other terms that must be
+ * scanned before a term is located and its frequency and position information
+ * may be processed. In a large index with user-entered query terms, query
+ * processing time is likely to be dominated not by term lookup but rather
+ * by the processing of frequency and positional data. In a small index
+ * or when many uncommon query terms are generated (e.g., by wildcard
+ * queries) term lookup may become a dominant cost.
+ *
+ * In particular, numUniqueTerms/interval terms are read into
+ * memory by an IndexReader, and, on average, interval/2 terms
+ * must be scanned for each random term access.
+ *
+ * @see #DEFAULT_TERM_INDEX_INTERVAL
+ */
+ public void setTermIndexInterval(int interval);
+
+ /**
+ * Expert: Return the interval between indexed terms.
+ *
+ * @see #setTermIndexInterval(int)
+ */
+ public int getTermIndexInterval();
+
+ /**
+ * Determines the minimal number of documents required before the buffered
+ * in-memory documents are merging and a new Segment is created.
+ * Since Documents are merged in a {@link org.apache.lucene.store.RAMDirectory},
+ * large value gives faster indexing. At the same time, mergeFactor limits
+ * the number of files open in a FSDirectory.
+ *
+ *
The default value is 10. + * + * + * @throws IllegalArgumentException if maxBufferedDocs is smaller than 2 + */ + public void setMaxBufferedDocs(int maxBufferedDocs); + + /** + * + * + * @see #setMaxBufferedDocs + */ + public int getMaxBufferedDocs(); + + /** + * The maximum number of terms that will be indexed for a single field in a + * document. This limits the amount of memory required for indexing, so that + * collections with very large files will not crash the indexing process by + * running out of memory.
+ * Note that this effectively truncates large documents, excluding from the + * index terms that occur further in the document. If you know your source + * documents are large, be sure to set this value high enough to accomodate + * the expected size. If you set it to Integer.MAX_VALUE, then the only limit + * is your memory, but you should anticipate an OutOfMemoryError. + * By default, no more than 10,000 terms will be indexed for a field. + */ + public void setMaxFieldLength(int maxFieldLength); + + /** + * + * + * @see #setMaxFieldLength + */ + public int getMaxFieldLength(); + + /** + * Determines the largest number of documents ever merged by addDocument(). + * Small values (e.g., less than 10,000) are best for interactive indexing, + * as this limits the length of pauses while indexing to a few seconds. + * Larger values are best for batched indexing and speedier searches. + * + *The default value is {@link Integer#MAX_VALUE}. + */ + public void setMaxMergeDocs(int maxMergeDocs); + + /** + * + * + * @see #setMaxMergeDocs + */ + public int getMaxMergeDocs(); + + /** + * Determines how often segment indices are merged by addDocument(). With + * smaller values, less RAM is used while indexing, and searches on + * unoptimized indices are faster, but indexing speed is slower. With larger + * values, more RAM is used during indexing, and while searches on unoptimized + * indices are slower, indexing is faster. Thus larger values (> 10) are best + * for batch index creation, and smaller values (< 10) for indices that are + * interactively maintained. + * + *
This must never be less than 2. The default value is 10.
+ */
+ public void setMergeFactor(int mergeFactor);
+
+ /**
+ *
+ *
+ * @see #setMergeFactor
+ */
+ public int getMergeFactor();
+
+ /**
+ * Sets the maximum time to wait for a write lock (in milliseconds).
+ */
+ public void setWriteLockTimeout(long writeLockTimeout);
+
+ /**
+ *
+ *
+ * @see #setWriteLockTimeout
+ */
+ public long getWriteLockTimeout();
+
+ /**
+ * Sets the maximum time to wait for a commit lock (in milliseconds).
+ */
+ public void setCommitLockTimeout(long commitLockTimeout);
+
+ /**
+ *
+ *
+ * @see #setCommitLockTimeout
+ */
+ public long getCommitLockTimeout();
+
+ /** Expert: Set the TokenSelector used to determine subset of tokens stored in term vectors.
+ * @param selector the term vector TokenSelector
+ */
+ public void setTermVectorTokenSelector(TokenSelector selector);
+
+ /** Expert: Set the TokenSelector used to determine subset of tokens stored in term vectors.
+ * @return the TokenSelector used to determine term vector tokens
+ */
+ public TokenSelector getTermVectorTokenSelector();
+
+ /** Expert: Set the TokenSelector used to determine subset of tokens for which positions are stored.
+ * (At least one position is always stored for each term in each doc to ensure the term stays in
+ * the index so long as any docs reference it)
+ * @param selector the positions TokenSelector
+ */
+ public void setPositionsTokenSelector(TokenSelector selector);
+
+ /** Expert: Set the TokenSelector used to determine subset of tokens for which freq and positions are stored..
+ * (At least one position is always stored for each term in each doc to ensure the term stays in
+ * the index so long as any docs reference it)
+ * @return the positions TokenSelector
+ */
+ public TokenSelector getPositionsTokenSelector();
+
+ /** If non-null, information about merges and a message when
+ * maxFieldLength is reached will be printed to this.
+ */
+ public void setInfoStream(PrintStream infoStream);
+
+ /**
+ *
+ *
+ * @see #setInfoStream
+ */
+ public PrintStream getInfoStream();
+
+}
Index: src/java/org/apache/lucene/index/DocumentWriter.java
===================================================================
--- src/java/org/apache/lucene/index/DocumentWriter.java (revision 414705)
+++ src/java/org/apache/lucene/index/DocumentWriter.java (working copy)
@@ -17,6 +17,7 @@
*/
import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenSelector;
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.document.Document;
@@ -35,6 +36,8 @@
final class DocumentWriter {
private Analyzer analyzer;
+ private TokenSelector termVectorTokenSelector;
+ private TokenSelector positionsTokenSelector;
private Directory directory;
private Similarity similarity;
private FieldInfos fieldInfos;
@@ -142,9 +145,9 @@
if (!field.isTokenized()) { // un-tokenized field
String stringValue = field.stringValue();
if(field.isStoreOffsetWithTermVector())
- addPosition(fieldName, stringValue, position++, new TermVectorOffsetInfo(offset, offset + stringValue.length()));
+ addPosition(fieldName, stringValue, position++, new TermVectorOffsetInfo(offset, offset + stringValue.length()), false, false);
else
- addPosition(fieldName, stringValue, position++, null);
+ addPosition(fieldName, stringValue, position++, null, false, false);
offset += stringValue.length();
length++;
} else
@@ -165,10 +168,16 @@
for (Token t = stream.next(); t != null; t = stream.next()) {
position += (t.getPositionIncrement() - 1);
- if(field.isStoreOffsetWithTermVector())
- addPosition(fieldName, t.termText(), position++, new TermVectorOffsetInfo(offset + t.startOffset(), offset + t.endOffset()));
- else
- addPosition(fieldName, t.termText(), position++, null);
+ boolean omittv = false, omitpos = false;
+ if (termVectorTokenSelector!=null && !termVectorTokenSelector.accept(field.name(), t))
+ omittv = true;
+ if (positionsTokenSelector !=null && !positionsTokenSelector. accept(field.name(), t))
+ omitpos = true;
+
+ addPosition(fieldName, t.termText(), position++,
+ field.isStoreOffsetWithTermVector() && !omittv ? new TermVectorOffsetInfo(offset + t.startOffset(), offset + t.endOffset())
+ : null,
+ omittv, omitpos);
lastToken = t;
if (++length > maxFieldLength) {
@@ -196,20 +205,24 @@
private final Term termBuffer = new Term("", ""); // avoid consing
- private final void addPosition(String field, String text, int position, TermVectorOffsetInfo offset) {
+ private final void addPosition(String field, String text, int position, TermVectorOffsetInfo offset,
+ boolean omitFromTermVector, boolean omitPosition) {
termBuffer.set(field, text);
//System.out.println("Offset: " + offset);
Posting ti = (Posting) postingTable.get(termBuffer);
if (ti != null) { // word seen before
int freq = ti.freq;
- if (ti.positions.length == freq) { // positions array is full
- int[] newPositions = new int[freq * 2]; // double size
- int[] positions = ti.positions;
- for (int i = 0; i < freq; i++) // copy old positions to new
- newPositions[i] = positions[i];
- ti.positions = newPositions;
+
+ if (!omitPosition) {
+ if (ti.positions.length == freq) { // positions array is full
+ int[] newPositions = new int[freq * 2]; // double size
+ int[] positions = ti.positions;
+ for (int i = 0; i < freq; i++) // copy old positions to new
+ newPositions[i] = positions[i];
+ ti.positions = newPositions;
+ }
+ ti.positions[freq] = position; // add new position
}
- ti.positions[freq] = position; // add new position
if (offset != null) {
if (ti.offsets.length == freq){
@@ -223,10 +236,12 @@
}
ti.offsets[freq] = offset;
}
- ti.freq = freq + 1; // update frequency
- } else { // word not seen before
+
+ if (!omitPosition)
+ ti.freq = freq + 1; // update frequency
+ } else { // word not seen before
Term term = new Term(field, text, false);
- postingTable.put(term, new Posting(term, position, offset));
+ postingTable.put(term, new Posting(term, position, offset, omitFromTermVector));
}
}
@@ -351,7 +366,7 @@
termVectorWriter.closeField();
}
}
- if (termVectorWriter != null && termVectorWriter.isFieldOpen()) {
+ if (termVectorWriter != null && termVectorWriter.isFieldOpen() && !posting.omitFromTermVector) {
termVectorWriter.addTerm(posting.term.text(), postingFreq, posting.positions, posting.offsets);
}
}
@@ -390,6 +405,16 @@
this.infoStream = infoStream;
}
+ /** If non-null, this will be used to select which tokens are stored in term vectors */
+ void setTermVectorTokenSelector(TokenSelector selector) {
+ this.termVectorTokenSelector = selector;
+ }
+
+ /** If non-null, this will be used to select which tokens have positions stored in the index. */
+ void setPositionsTokenSelector(TokenSelector selector) {
+ this.positionsTokenSelector = selector;
+ }
+
}
final class Posting { // info about a Term in a doc
@@ -397,17 +422,17 @@
int freq; // its frequency in doc
int[] positions; // positions it occurs at
TermVectorOffsetInfo [] offsets;
+ boolean omitFromTermVector; // if true, omit from term vector
- Posting(Term t, int position, TermVectorOffsetInfo offset) {
+ Posting(Term t, int position, TermVectorOffsetInfo offset, boolean omitFromTermVector) {
term = t;
freq = 1;
positions = new int[1];
positions[0] = position;
- if(offset != null){
- offsets = new TermVectorOffsetInfo[1];
- offsets[0] = offset;
+ if(offset != null) {
+ offsets = new TermVectorOffsetInfo[1];
+ offsets[0] = offset;
}
- else
- offsets = null;
+ this.omitFromTermVector = omitFromTermVector;
}
}
Index: src/java/org/apache/lucene/index/ParallelWriter.java
===================================================================
--- src/java/org/apache/lucene/index/ParallelWriter.java (revision 0)
+++ src/java/org/apache/lucene/index/ParallelWriter.java (revision 0)
@@ -0,0 +1,345 @@
+/*
+ * ParallelWriter.java
+ *
+ * Created on April 28, 2006, 7:07 PM
+ *
+ */
+
+package org.apache.lucene.index;
+
+import java.io.IOException;
+import java.io.PrintStream;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenSelector;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.search.Similarity;
+import org.apache.lucene.store.Directory;
+
+/**
+ * ParallelWriter is a companion to ParallelReader, although as with IndexWriter it only supports indexes stored in a Directory.
+ * The interface is at the field level. A map from directories to lists of fields is provided to create the ParallelWriter,
+ * which then creates an IndexWriter for each specified directory and operates on each field of a document using the IndexWriter
+ * for the directory to which that field is mapped. This mapping allows an application to configure its use of parallel sub-
+ * indexes independently from the rest of its processing.
+ *
+ * This implementation single-threads calls to addDocument(), but does the sub-document writes in parallel. Users of this class
+ * must ensure that the ParallelReader is never reopened while adding a new document, and must deal with recovery if exceptions
+ * occur while adding a document.
+ *
+ * @author Chuck Williams
+ */
+public class ParallelWriter implements Writable {
+
+ IndexWriter[] writers; // All IndexWriters
+ IOException exception; // If any writer gets an exception, this is stored here (only one needed)
+ Map