diff --git a/lucene/core/src/java/org/apache/lucene/index/package-info.java b/lucene/core/src/java/org/apache/lucene/index/package-info.java index f5a86d1..bca6eba 100644 --- a/lucene/core/src/java/org/apache/lucene/index/package-info.java +++ b/lucene/core/src/java/org/apache/lucene/index/package-info.java @@ -6,7 +6,7 @@ * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * - * http://www.apache.org/licenses/LICENSE-2.0 + * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, @@ -17,34 +17,112 @@ /** * Code to maintain and access indices. - * *

Table Of Contents

- *
    - *
  1. Postings APIs - * - *
    2. - *
  2. Index Statistics - * - *
    3. - *
+ *
    + *
  1. Index APIs + * + * + * + *
    2. + *
  2. Field types + * + *
    3. + *
  3. Postings APIs + * + *
    4. + *
  4. Index Statistics + * + *
    5. + *
+ * + *

Index APIs

+ + * + *

IndexWriter

+ + *

{@link IndexWriter} is used to create an index, and to add, update and delete documents. The + * IndexWriter class is thread safe, and enforces a single instance per index. Creating an + * IndexWriter creates a new index or opens an existing index for writing, in a {@link + * org.apache.lucene.store.Directory}, depending on the configuration in {@link + * IndexWriterConfig}. A Directory is an abstraction that typically represents a local file-system + * directory (see various implementations of {@link org.apache.lucene.store.FSDirectory}), but it + * may also stand for some other storage, such as RAM or a distributed filesystem.

+ + * + *

IndexReader

+ + *

{@link IndexReader} is used to read data from the index, and supports searching. Many + * thread-safe readers may be open concurrently with a single (or no) writer. Each reader maintains + * a consistent "point in time" view of an index and must be explicitly refreshed in order to + * incorporate writes that may occur after it is opened.

+ + * + *

Segments and docids

+ + *

Lucene's index is composed of segments, each of which contains a subset of all the documents + * in the index, and is a complete searchable index in itself, over that subset. As documents are + * written to the index, new segments are created and flushed to directory storage. Segments are + * immutable; updates and deletions may only create new segments and do not modify existing + * ones. Over time, the writer merges groups of smaller segments into single larger ones in order to + * maintain an index that is efficient to search, and to reclaim dead space left behind by deleted + * (and updated) documents.

+ + *

Each document is identified by a 32-bit number, its "docid," and is composed of a collection + * of Field values of diverse types (postings, stored fields, doc values, and points). Docids come + * in two flavors: global and per-segment. A document's global docid is just the sum of its + * per-segment docid and that segment's base docid offset. External, high-level APIs only handle + * global docids, but internal APIs that reference a {@link LeafReader}, which is a reader for a + * single segment, deal in per-segment docids.

+ * + * + *

Field Types

+ * + *

Lucene supports a variety of different document field data structures. Lucene's core, the + * inverted index, is comprised of "postings." The postings, with their term dictionary, can be + * thought of as a map that provides efficient lookup given a {@link Term} (roughly, a word or + * token), to (the ordered list of) {@link Document}s containing that Term. Postings do not provide + * any way of retrieving terms given a document, short of scanning the entire index.

+ * + *

Stored fields are essentially the opposite of postings, providing efficient retrieval of field + * values given a docid. All stored field values for a document are stored together in a + * block. Different types of stored field provide high-level datatypes such as strings and numbers + * on top of the underlying bytes. Stored field values are usually retrieved by the searcher using + * an implementation of {@link StoredFieldVisitor}.

+ + *

{@link DocValues} fields are what are sometimes referred to as columnar, or column-stride fields, by + * analogy to relational database terminology, in which documents are considered as rows, and + * fields, columns. DocValues fields store values per-field: a value for every document is held in a + * single data structure, providing for rapid, random-access lookup of a field-value given a + * docid. These fields are used for efficient value-based sorting, and for faceting and can be + * retrieved efficiently, but they are not useful for filtering.

+ + *

{@link PointValues} represent numeric values using a kd-tree data structure. Efficient 1- and + * higher dimensional implementations make these the choice for numeric range and interval queries, + * and geo-spatial queries.

+ * *

Postings APIs

* *

- * Fields + * Fields *

*

- * {@link org.apache.lucene.index.Fields} is the initial entry point into the + * {@link org.apache.lucene.index.Fields} is the initial entry point into the * postings APIs, this can be obtained in several ways: * 

  * // access indexed fields for an index segment
@@ -63,7 +141,7 @@
  *
* *

- * Terms + * Terms *

*

* {@link org.apache.lucene.index.Terms} represents the collection of terms @@ -128,10 +206,10 @@ * System.out.println(docid); * int freq = postings.freq(); * for (int i = 0; i < freq; i++) { - * System.out.println(postings.nextPosition()); - * System.out.println(postings.startOffset()); - * System.out.println(postings.endOffset()); - * System.out.println(postings.getPayload()); + * System.out.println(postings.nextPosition()); + * System.out.println(postings.startOffset()); + * System.out.println(postings.endOffset()); + * System.out.println(postings.getPayload()); * } * } * @@ -139,101 +217,101 @@ *

Index Statistics

* *

- * Term statistics + * Term statistics *

- * + * * *

- * Field statistics + * Field statistics *

- * + * * *

- * Segment statistics + * Segment statistics *

- * + * * *

- * Document statistics + * Document statistics *

*

* Document statistics are available during the indexing process for an indexed field: typically * a {@link org.apache.lucene.search.similarities.Similarity} implementation will store some * of these values (possibly in a lossy way), into the normalization value for the document in * its {@link org.apache.lucene.search.similarities.Similarity#computeNorm} method. - *

+ * *

* Additional user-supplied statistics can be added to the document as DocValues fields and * accessed via {@link org.apache.lucene.index.LeafReader#getNumericDocValues}.