Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Terms dict index needs to store large RAM resident arrays of ints, but, because their size is bound & variable (depending on the segment/docs), we should used packed ints for them.

      1. LUCENE-2321.patch
        42 kB
        Michael McCandless
      2. LUCENE-2321.patch
        27 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        First cut – it has some limitations though:

        • Cannot handle termsIndexInterval yet (forces this to 1)
        • Loads all term data as a single byte[] – this may be a problem, eg it means your total bytes for indexed terms cannot exceed 2.1B; it also may fail to allocate due to memory fragmentation on 32 bit JREs. So I think we have to make this paged.

        Also found & fixed some small bugs in packed ints.

        Show
        Michael McCandless added a comment - First cut – it has some limitations though: Cannot handle termsIndexInterval yet (forces this to 1) Loads all term data as a single byte[] – this may be a problem, eg it means your total bytes for indexed terms cannot exceed 2.1B; it also may fail to allocate due to memory fragmentation on 32 bit JREs. So I think we have to make this paged. Also found & fixed some small bugs in packed ints.
        Hide
        Michael McCandless added a comment -

        New rev – I think this one's ready to commit.

        It handles indexDivisor, and pages the index term bytes allocations.

        Show
        Michael McCandless added a comment - New rev – I think this one's ready to commit. It handles indexDivisor, and pages the index term bytes allocations.
        Hide
        Michael McCandless added a comment -

        Flex is winding down

        Show
        Michael McCandless added a comment - Flex is winding down

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development