Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5688

NumericDocValues fields with sparse data can be compressed better

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I ran into this problem where I had a dynamic field in Solr and indexed data into lots of fields. For each field only a few documents had actual values and the remaining documents the default value ( 0 ) got indexed. Now when I merge segments, the index size jumps up.

      For example I have 10 segments - Each with 1 DV field. When I merge segments into 1 that segment will contain all 10 DV fields with lots if 0s.

      This was the motivation behind trying to come up with a compression for a use case like this.

        Attachments

        1. LUCENE-5688.patch
          26 kB
          Varun Thacker
        2. LUCENE-5688.patch
          25 kB
          Varun Thacker
        3. LUCENE-5688.patch
          10 kB
          Varun Thacker

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                varunthacker Varun Thacker
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: