Lucene - Core
  1. Lucene - Core
  2. LUCENE-4928

Compressed stored fields: make the maximum number of docs in a chunk configurable

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      When documents are very small (a few bytes), there can be so many of them in a single chunk that merging can become very slow. Making the maximum number of documents per chunk configurable could help.

        Activity

        Hide
        Robert Muir added a comment -

        +1, i imagine it would also speed up stored fields access in such a case (at the expense of compression, but in such a case you are using so little disk anyway that I think its worth it).

        Show
        Robert Muir added a comment - +1, i imagine it would also speed up stored fields access in such a case (at the expense of compression, but in such a case you are using so little disk anyway that I think its worth it).
        Hide
        Adrien Grand added a comment -

        I'm looking at the term vectors format, and it can't have a configurable number of documents per chunk without changing the format (it would need to store the max number of documents per chunk to be able at merging time to decide on whether it can bulk-merge the next chunk). So for now I think we can just have a hard limit and make it configurable in the future if we have a need for it?

        Show
        Adrien Grand added a comment - I'm looking at the term vectors format, and it can't have a configurable number of documents per chunk without changing the format (it would need to store the max number of documents per chunk to be able at merging time to decide on whether it can bulk-merge the next chunk). So for now I think we can just have a hard limit and make it configurable in the future if we have a need for it?
        Hide
        Adrien Grand added a comment -

        Proposed patch.

        Show
        Adrien Grand added a comment - Proposed patch.
        Hide
        Robert Muir added a comment -

        +1, lets start simple.

        Show
        Robert Muir added a comment - +1, lets start simple.
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Adrien Grand
            Reporter:
            Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development