Lucene - Core
  1. Lucene - Core
  2. LUCENE-1826

All Tokenizer implementations should have constructors that take AttributeSource and AttributeFactory

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.9
    • Fix Version/s: 2.9
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I have a TokenStream implementation that joins together multiple sub TokenStreams (i then do additional filtering on top of this, so i can't just have the indexer do the merging)

      in 2.4, this worked fine.
      once one sub stream was exhausted, i just started using the next stream

      however, in 2.9, this is very difficult, and requires copying Term buffers for every token being aggregated

      however, if all the sub TokenStreams share the same AttributeSource, and my "concat" TokenStream shares the same AttributeSource, this goes back to being very simple (and very efficient)

      So for example, i would like to see the following constructor added to StandardTokenizer:

        public StandardTokenizer(AttributeSource source, Reader input, boolean replaceInvalidAcronym) {
          super(source);
          ...
        }
      

      would likewise want similar constructors added to all Tokenizer sub classes provided by lucene

      1. lucene-1826.patch
        26 kB
        Michael Busch

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Michael Busch
            Reporter:
            Tim Smith
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development