Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2321

use packed ints for the terms dict index

        Details

        • Type: Improvement
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 4.0-ALPHA
        • Fix Version/s: 4.0-ALPHA
        • Component/s: core/index
        • Labels:
          None
        • Lucene Fields:
          New

          Description

          Terms dict index needs to store large RAM resident arrays of ints, but, because their size is bound & variable (depending on the segment/docs), we should used packed ints for them.

          1. Text File
            LUCENE-2321.patch
            42 kB
            Michael McCandless
          2. Text File
            LUCENE-2321.patch
            27 kB
            Michael McCandless

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            mikemccand Michael McCandless added a comment -

            First cut – it has some limitations though:

            • Cannot handle termsIndexInterval yet (forces this to 1)
            • Loads all term data as a single byte[] – this may be a problem, eg it means your total bytes for indexed terms cannot exceed 2.1B; it also may fail to allocate due to memory fragmentation on 32 bit JREs. So I think we have to make this paged.

            Also found & fixed some small bugs in packed ints.

            Show
            mikemccand Michael McCandless added a comment - First cut – it has some limitations though: Cannot handle termsIndexInterval yet (forces this to 1) Loads all term data as a single byte[] – this may be a problem, eg it means your total bytes for indexed terms cannot exceed 2.1B; it also may fail to allocate due to memory fragmentation on 32 bit JREs. So I think we have to make this paged. Also found & fixed some small bugs in packed ints.
            Hide
            Permalink
            mikemccand Michael McCandless added a comment -

            New rev – I think this one's ready to commit.

            It handles indexDivisor, and pages the index term bytes allocations.

            Show
            mikemccand Michael McCandless added a comment - New rev – I think this one's ready to commit. It handles indexDivisor, and pages the index term bytes allocations.
            Hide
            Permalink
            mikemccand Michael McCandless added a comment -

            Flex is winding down

            Show
            mikemccand Michael McCandless added a comment - Flex is winding down

              People

              • Assignee:
                mikemccand Michael McCandless
                Reporter:
                mikemccand Michael McCandless
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development