Solr
  1. Solr
  2. SOLR-1312

BufferedTokenStream should use new Lucene 2.9 TokenStream API

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.4
    • Fix Version/s: None
    • Component/s: Schema and Analysis
    • Labels:
      None

      Description

      Since Solr 1.4 will be using Lucene 2.9, the Solr TokenFilters should probably be updated to use the Lucene 2.9 TokenStream API. This issue is to put BufferedTokenStream on the list of Filters that need updating.

        Issue Links

          Activity

          Tom Burton-West created issue -
          Tom Burton-West made changes -
          Field Original Value New Value
          Summary BufferedTokenStream should use new Lucene 2.9 tokenstream API BufferedTokenStream should use new Lucene 2.9 TokenStream API
          Priority Major [ 3 ] Minor [ 4 ]
          Jason Rutherglen made changes -
          Link This issue blocks SOLR-908 [ SOLR-908 ]
          Hide
          Jason Rutherglen added a comment -

          Tom, I'm going to put a patch together for this.

          Show
          Jason Rutherglen added a comment - Tom, I'm going to put a patch together for this.
          Hide
          Grant Ingersoll added a comment -

          FYI: I can't find it at the moment, but I think Yonik is doing a bunch of work on the Analysis stuff for Lucene 2.9 uptake in Solr, so it may be taken care of.

          Show
          Grant Ingersoll added a comment - FYI: I can't find it at the moment, but I think Yonik is doing a bunch of work on the Analysis stuff for Lucene 2.9 uptake in Solr, so it may be taken care of.
          Hide
          Robert Muir added a comment -

          Hello, I recommend we cancel this issue.

          No Solr tokenstreams extend this BufferedTokenStream API anymore, as it is bound to Token and does not support reuse.
          Currently this class is marked deprecated in trunk, with a backwards compatibility layer.

          If we think that an API like this is useful, we should make a new BufferedTokenStream-like API that uses AttributeSource
          instead of Token, but this API would not support reuse and would not be very performant, as it would have to use
          cloneAttributes() and copyTo() instead of captureState() and restoreState()

          Show
          Robert Muir added a comment - Hello, I recommend we cancel this issue. No Solr tokenstreams extend this BufferedTokenStream API anymore, as it is bound to Token and does not support reuse. Currently this class is marked deprecated in trunk, with a backwards compatibility layer. If we think that an API like this is useful, we should make a new BufferedTokenStream-like API that uses AttributeSource instead of Token, but this API would not support reuse and would not be very performant, as it would have to use cloneAttributes() and copyTo() instead of captureState() and restoreState()
          Gavin made changes -
          Link This issue blocks SOLR-908 [ SOLR-908 ]
          Gavin made changes -
          Link This issue is depended upon by SOLR-908 [ SOLR-908 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Tom Burton-West
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development