Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6973

Improve TeeSinkTokenFilter


    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 5.5, 6.0
    • None
    • None
    • New, Patch Available


      TeeSinkTokenFilter can be improved in several ways, as it's written today:

      The most major one is removing SinkFilter which just doesn't work and is confusing. E.g., if you set a SinkFilter which filters tokens, the attributes on the stream such as PositionIncrementAttribute are not updated. Also, if you update any attribute on the stream, you affect other SinkStreams ... It's best if we remove this confusing class, and let consumers reuse existing TokenFilters by chaining them to the sink stream.

      After we do that, we can make all the cached states a single (immutable) list, which is shared between all the sink streams, so we don't need to keep many references around, and also deal with WeakReference.

      Besides that there are some other minor improvements to the code that will come after we clean up this class.

      From a backwards-compatibility standpoint, I don't think that SinkFilter is actually used anywhere (since it just ... confusing and doesn't work as expected), and therefore I believe it won't affect anyone. If however someone did implement a SinkFilter, it should be trivial to convert it to a TokenFilter and chain it to the SinkStream.


        1. LUCENE-6973.patch
          51 kB
          Shai Erera
        2. LUCENE-6973.patch
          49 kB
          Shai Erera
        3. LUCENE-6973.patch
          48 kB
          Shai Erera
        4. LUCENE-6973.patch
          63 kB
          Shai Erera
        5. LUCENE-6973.patch
          64 kB
          Shai Erera
        6. LUCENE-6973.patch
          64 kB
          Shai Erera
        7. LUCENE-6973.patch
          65 kB
          Shai Erera
        8. LUCENE-6973.patch
          65 kB
          Shai Erera



            shaie Shai Erera
            shaie Shai Erera
            0 Vote for this issue
            2 Start watching this issue