Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      With Analyzer now using TokenStreamComponents, theres no reason for Analysis consumers to use tokenStream() (it just gives bad performance). Consequently all consumers will be moved over to using reusableTokenStream(). The only challenge here is that reusableTokenStream throws an IOException which many consumers are not rigged to deal with.

      Once all consumers have been moved, we can rename reusableTokenStream() back to tokenStream().

      1. LUCENE-3455.patch
        12 kB
        Chris Male
      2. LUCENE-3455-test-consumers.patch
        43 kB
        Chris Male
      3. LUCENE-3455-test-consumers.patch
        37 kB
        Chris Male

        Activity

        Hide
        Robert Muir added a comment -

        +1, there is a lot of crazy code around this area, consumers catching the exception from reusableTokenStream() and falling back to tokenStream() and other silly things.

        Show
        Robert Muir added a comment - +1, there is a lot of crazy code around this area, consumers catching the exception from reusableTokenStream() and falling back to tokenStream() and other silly things.
        Hide
        Chris Male added a comment -

        Patch which converts all test consumers over to using reusableTokenStream.

        Included is a fix for PatternAnalyzer, where I had missed re-creating the Matcher which prevented reuse.

        I'm going to commit this shortly and then attack production code.

        Show
        Chris Male added a comment - Patch which converts all test consumers over to using reusableTokenStream. Included is a fix for PatternAnalyzer, where I had missed re-creating the Matcher which prevented reuse. I'm going to commit this shortly and then attack production code.
        Hide
        Chris Male added a comment -

        Fixed some failing tests. Now everything is passing.

        Committing shortly.

        Show
        Chris Male added a comment - Fixed some failing tests. Now everything is passing. Committing shortly.
        Hide
        Chris Male added a comment -

        Committed revision 1175670.

        Show
        Chris Male added a comment - Committed revision 1175670.
        Hide
        Chris Male added a comment -

        Patch which converts the last consumers over to using reusableTokenStream().

        Couple of changes to the classic QP included in this. Rather than silently returning null on an IOException, a ParseException is now thrown. Equally, where before the IOException was simply ignored, an Exception is thrown.

        Going to commit this shortly and then change reusableTokenStream() to tokenStream().

        Show
        Chris Male added a comment - Patch which converts the last consumers over to using reusableTokenStream(). Couple of changes to the classic QP included in this. Rather than silently returning null on an IOException, a ParseException is now thrown. Equally, where before the IOException was simply ignored, an Exception is thrown. Going to commit this shortly and then change reusableTokenStream() to tokenStream().
        Hide
        Chris Male added a comment -

        Committed revision 1176191.

        Show
        Chris Male added a comment - Committed revision 1176191.

          People

          • Assignee:
            Chris Male
            Reporter:
            Chris Male
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development