Lucene - Core
  1. Lucene - Core
  2. LUCENE-2949

FastVectorHighlighter FieldTermStack could likely benefit from using TermVectorMapper

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.0.3, 4.0-ALPHA
    • Fix Version/s: None
    • Component/s: None

      Description

      Based on my reading of the FieldTermStack constructor that loads the vector from disk, we could probably save a bunch of time and memory by using the TermVectorMapper callback mechanism instead of materializing the full array of terms into memory and then throwing most of them out.

      1. LUCENE-2949.patch
        3 kB
        Koji Sekiguchi

        Activity

        Hide
        Adrien Grand added a comment -

        There is no TermVectorMapper anymore.

        Show
        Adrien Grand added a comment - There is no TermVectorMapper anymore.
        Hide
        Hoss Man added a comment -

        Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently.

        email notification suppressed to prevent mass-spam
        psuedo-unique token identifying these issues: hoss20120321nofix36

        Show
        Hoss Man added a comment - Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently. email notification suppressed to prevent mass-spam psuedo-unique token identifying these issues: hoss20120321nofix36
        Hide
        Koji Sekiguchi added a comment -

        Cool, I like the idea! But I don't have much time to try it now, I'll unassign myself.

        Show
        Koji Sekiguchi added a comment - Cool, I like the idea! But I don't have much time to try it now, I'll unassign myself.
        Hide
        Simon Willnauer added a comment -

        Koji, I wonder if we can make use of:

        public abstract void setExpectations(String field, int numTerms, boolean storeOffsets, boolean storePositions);
        

        we could actually allocate an array for TermInfo instead of a LinkedList which creates tons of object. I'd even go further and say we allocate a parallel array ie. one array for start & end offset, position, term (ByteBlockPool maybe even?) this would reduce the num of objects dramatically.

        Show
        Simon Willnauer added a comment - Koji, I wonder if we can make use of: public abstract void setExpectations( String field, int numTerms, boolean storeOffsets, boolean storePositions); we could actually allocate an array for TermInfo instead of a LinkedList which creates tons of object. I'd even go further and say we allocate a parallel array ie. one array for start & end offset, position, term (ByteBlockPool maybe even?) this would reduce the num of objects dramatically.
        Hide
        Koji Sekiguchi added a comment -

        Thanks Simon for awaking me on this! I'll commit if nobody objects.

        Show
        Koji Sekiguchi added a comment - Thanks Simon for awaking me on this! I'll commit if nobody objects.
        Hide
        Simon Willnauer added a comment -

        anybody working on this? koji, if you are not going to work on this can you unassign you from the issue?

        Show
        Simon Willnauer added a comment - anybody working on this? koji, if you are not going to work on this can you unassign you from the issue?
        Hide
        Mike Sokolov added a comment -

        This looks like the same issue as LUCENENET-350?

        Show
        Mike Sokolov added a comment - This looks like the same issue as LUCENENET-350 ?
        Hide
        Robert Muir added a comment -

        bulk move 3.2 -> 3.3

        Show
        Robert Muir added a comment - bulk move 3.2 -> 3.3
        Hide
        Grant Ingersoll added a comment -

        I haven't looked at the patch yet, but the setExpectations is basically there in case you wish to pre allocate any structures.

        Show
        Grant Ingersoll added a comment - I haven't looked at the patch yet, but the setExpectations is basically there in case you wish to pre allocate any structures.
        Hide
        Koji Sekiguchi added a comment -

        A first draft patch. All FVH tests pass.

        As it is my first experience of TermVectorMapper, feedback would be appreciated. For example, I left setExpectations() empty, but I'm not sure it is correct.

        Show
        Koji Sekiguchi added a comment - A first draft patch. All FVH tests pass. As it is my first experience of TermVectorMapper, feedback would be appreciated. For example, I left setExpectations() empty, but I'm not sure it is correct.

          People

          • Assignee:
            Unassigned
            Reporter:
            Grant Ingersoll
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development