Solr
  1. Solr
  2. SOLR-3504

Clearly document the limit for the maximum number of documents in a single index

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.6
    • Fix Version/s: None
    • Component/s: update
    • Labels:
      None

      Description

      Although the actual limit to the number of documents supported by a Solr implementation depends on the number of shards, unless the user is intimately familiar with the implementation of Lucene, they may not realize that a single Solr index (single shard, single core) is limited to approximately 2.14 billion documents regardless of their processing power or memory. This limit should be clearly documented for the Solr user.

      Granted, users should be strongly discouraged from attempting to create a single, unsharded index of that size, but they certainly should have to find out about the Lucene limit by accident.

      A subsequent issue will recommend that Solr detect and appropriately report to the user when and if this limit is hit.

        Activity

        Hide
        Jack Krupansky added a comment -

        Solr has a number of interfaces for adding documents, such as Update XML, Update CSV, SolrCell, Data Import Handler, SolrCloud, etc. Generally, each has a wiki page, to which the following section should be added:

        "Limitations

        Although a Solr implementation can scale into the billions of documents by using a number of shards, each individual shard or Solr core is limited by the Lucene limit for an index which is approximately 2.14 billion documents (2,147,483,647 to be exact) in the current implementation of Lucene. In practice, it is unlikely that such a large number of documents would fit and perform well in a single index. In extreme cases it may be possible, but in no case can the number of documents in a single index exceed that number."

        This limitation could also be added to the Solr tutorial page.

        There are probably a few other locations in the Solr docs when this limitation should be noted.

        Show
        Jack Krupansky added a comment - Solr has a number of interfaces for adding documents, such as Update XML, Update CSV, SolrCell, Data Import Handler, SolrCloud, etc. Generally, each has a wiki page, to which the following section should be added: "Limitations Although a Solr implementation can scale into the billions of documents by using a number of shards, each individual shard or Solr core is limited by the Lucene limit for an index which is approximately 2.14 billion documents (2,147,483,647 to be exact) in the current implementation of Lucene. In practice, it is unlikely that such a large number of documents would fit and perform well in a single index. In extreme cases it may be possible, but in no case can the number of documents in a single index exceed that number." This limitation could also be added to the Solr tutorial page. There are probably a few other locations in the Solr docs when this limitation should be noted.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jack Krupansky
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development