Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1826

All Tokenizer implementations should have constructors that take AttributeSource and AttributeFactory

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9
    • Fix Version/s: 2.9
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I have a TokenStream implementation that joins together multiple sub TokenStreams (i then do additional filtering on top of this, so i can't just have the indexer do the merging)

      in 2.4, this worked fine.
      once one sub stream was exhausted, i just started using the next stream

      however, in 2.9, this is very difficult, and requires copying Term buffers for every token being aggregated

      however, if all the sub TokenStreams share the same AttributeSource, and my "concat" TokenStream shares the same AttributeSource, this goes back to being very simple (and very efficient)

      So for example, i would like to see the following constructor added to StandardTokenizer:

        public StandardTokenizer(AttributeSource source, Reader input, boolean replaceInvalidAcronym) {
          super(source);
          ...
        }
      

      would likewise want similar constructors added to all Tokenizer sub classes provided by lucene

        Attachments

          Activity

            People

            • Assignee:
              michaelbusch Michael Busch
              Reporter:
              tsmith Tim Smith

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment