[LUCENE-3844] Deprecate Token class and remove in trunk

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5
• Fix Version/s: 4.9, 6.0
• Component/s: core/other, modules/analysis
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Uwe Schindler
• Created: 03/Mar/12 23:10
• Updated: 28/Aug/22 13:10

Description

After issues like LUCENE-3843, introducing new attributes, we should remove Token class in trunk, as it leads to code that ignores those new attributes (like PositionLengthAttribute, ScriptAttribute, KeywordAttribute,...). If you want a holder for all Attributes a TokenStream, use TS.cloneAttributes().

Issue Links

• is related to: LUCENE-3843 implement PositionLengthAttribute for all tokenstreams where its appropriate