Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5914

More options for stored fields compression

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.0, 6.0
    • None
    • None
    • New

    Description

      Since we added codec-level compression in Lucene 4.1 I think I got about the same amount of users complaining that compression was too aggressive and that compression was too light.

      I think it is due to the fact that we have users that are doing very different things with Lucene. For example if you have a small index that fits in the filesystem cache (or is close to), then you might never pay for actual disk seeks and in such a case the fact that the current stored fields format needs to over-decompress data can sensibly slow search down on cheap queries.

      On the other hand, it is more and more common to use Lucene for things like log analytics, and in that case you have huge amounts of data for which you don't care much about stored fields performance. However it is very frustrating to notice that the data that you store takes several times less space when you gzip it compared to your index although Lucene claims to compress stored fields.

      For that reason, I think it would be nice to have some kind of options that would allow to trade speed for compression in the default codec.

      Attachments

        1. LUCENE-5914.patch
          14 kB
          Robert Muir
        2. LUCENE-5914.patch
          14 kB
          Robert Muir
        3. LUCENE-5914.patch
          7 kB
          Robert Muir
        4. LUCENE-5914.patch
          211 kB
          Adrien Grand
        5. LUCENE-5914.patch
          199 kB
          Robert Muir
        6. LUCENE-5914.patch
          200 kB
          Robert Muir
        7. LUCENE-5914.patch
          100 kB
          Adrien Grand
        8. LUCENE-5914.patch
          61 kB
          Adrien Grand

        Issue Links

          Activity

            People

              rcmuir Robert Muir
              jpountz Adrien Grand
              Votes:
              5 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: