Lucene - Core
  1. Lucene - Core
  2. LUCENE-5384

Analysis overview could mention clearAttributes and end

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      It would be helpful to tokenizer implementors for the analysis package overview to mention more things. I'll supply a patch.

        Activity

        Hide
        Benson Margulies added a comment -

        https://github.com/apache/lucene-solr/pull/12 contains some more documentation.

        Yes, this is offered under the terms of the Apache license, in case anyone is still uncertain as to the relationship of github pull requests to the AL.

        Show
        Benson Margulies added a comment - https://github.com/apache/lucene-solr/pull/12 contains some more documentation. Yes, this is offered under the terms of the Apache license, in case anyone is still uncertain as to the relationship of github pull requests to the AL.
        Hide
        Uwe Schindler added a comment -

        Hi Benson,
        thank you for the patch! I would suggest to change the sentence that mentions clearAttributes() to be more clear, some ideas:

        • every Tokenizer must call clearAttributes (as we have already in the text)
        • every TokenFilter that inserts new tokens into the stream must also call clearAttributes(). Alternatively use captureState()/restoreState() to clone the previous token and modify it afterwards (this is the recommended way to insert tokens from TokenFilters).

        Uwe

        Show
        Uwe Schindler added a comment - Hi Benson, thank you for the patch! I would suggest to change the sentence that mentions clearAttributes() to be more clear, some ideas: every Tokenizer must call clearAttributes (as we have already in the text) every TokenFilter that inserts new tokens into the stream must also call clearAttributes(). Alternatively use captureState()/restoreState() to clone the previous token and modify it afterwards (this is the recommended way to insert tokens from TokenFilters). Uwe
        Hide
        ASF subversion and git services added a comment -

        Commit 1555907 from Robert Muir in branch 'dev/trunk'
        [ https://svn.apache.org/r1555907 ]

        LUCENE-5384: Add some analysis api tips to the package.html (closes #12)

        Show
        ASF subversion and git services added a comment - Commit 1555907 from Robert Muir in branch 'dev/trunk' [ https://svn.apache.org/r1555907 ] LUCENE-5384 : Add some analysis api tips to the package.html (closes #12)
        Hide
        Robert Muir added a comment -

        Thanks Benson! Really appreciate the doc contributions.

        Show
        Robert Muir added a comment - Thanks Benson! Really appreciate the doc contributions.
        Hide
        ASF subversion and git services added a comment -

        Commit 1555914 from Robert Muir in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1555914 ]

        LUCENE-5384: Add some analysis api tips to the package.html (closes #12)

        Show
        ASF subversion and git services added a comment - Commit 1555914 from Robert Muir in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1555914 ] LUCENE-5384 : Add some analysis api tips to the package.html (closes #12)

          People

          • Assignee:
            Benson Margulies
            Reporter:
            Benson Margulies
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development