Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-4928

Compressed stored fields: make the maximum number of docs in a chunk configurable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      When documents are very small (a few bytes), there can be so many of them in a single chunk that merging can become very slow. Making the maximum number of documents per chunk configurable could help.

        Activity

        Hide
        rcmuir Robert Muir added a comment -

        +1, i imagine it would also speed up stored fields access in such a case (at the expense of compression, but in such a case you are using so little disk anyway that I think its worth it).

        Show
        rcmuir Robert Muir added a comment - +1, i imagine it would also speed up stored fields access in such a case (at the expense of compression, but in such a case you are using so little disk anyway that I think its worth it).
        Hide
        jpountz Adrien Grand added a comment -

        I'm looking at the term vectors format, and it can't have a configurable number of documents per chunk without changing the format (it would need to store the max number of documents per chunk to be able at merging time to decide on whether it can bulk-merge the next chunk). So for now I think we can just have a hard limit and make it configurable in the future if we have a need for it?

        Show
        jpountz Adrien Grand added a comment - I'm looking at the term vectors format, and it can't have a configurable number of documents per chunk without changing the format (it would need to store the max number of documents per chunk to be able at merging time to decide on whether it can bulk-merge the next chunk). So for now I think we can just have a hard limit and make it configurable in the future if we have a need for it?
        Hide
        jpountz Adrien Grand added a comment -

        Proposed patch.

        Show
        jpountz Adrien Grand added a comment - Proposed patch.
        Hide
        rcmuir Robert Muir added a comment -

        +1, lets start simple.

        Show
        rcmuir Robert Muir added a comment - +1, lets start simple.
        Hide
        thetaphi Uwe Schindler added a comment -

        Closed after release.

        Show
        thetaphi Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            jpountz Adrien Grand
            Reporter:
            jpountz Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development