Lucene - Core
  1. Lucene - Core
  2. LUCENE-1699

Field tokenStream should be usable with stored fields.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Field.tokenStream should be usable for indexing even for stored values. Useful for many types of pre-analyzed values (text/numbers, etc)
      http://search.lucidimagination.com/search/document/902bad4eae20bdb8/field_tokenstreamvalue

      1. LUCENE-1699.patch
        14 kB
        Yonik Seeley
      2. LUCENE-1699.patch
        5 kB
        Yonik Seeley

        Issue Links

          Activity

          Hide
          Yonik Seeley added a comment -

          incomplete patch for feedback - needs tests and further work to allow binary field values with token streams (I assume this would be desirable for numeric fields).

          Show
          Yonik Seeley added a comment - incomplete patch for feedback - needs tests and further work to allow binary field values with token streams (I assume this would be desirable for numeric fields).
          Hide
          Mark Miller added a comment -

          Another 2.9 huh? You own it then

          Show
          Mark Miller added a comment - Another 2.9 huh? You own it then
          Hide
          Yonik Seeley added a comment -

          Watch out - there's a new sheriff in town!

          Show
          Yonik Seeley added a comment - Watch out - there's a new sheriff in town!
          Hide
          Mark Miller added a comment -

          This group could develop hard for the rest of the year no problem by the looks. Someones got to herd these cats. Since we last semi decided it would be a good idea to release 2.9, we have rewritten half the core code base

          Show
          Mark Miller added a comment - This group could develop hard for the rest of the year no problem by the looks. Someones got to herd these cats. Since we last semi decided it would be a good idea to release 2.9, we have rewritten half the core code base
          Hide
          Michael McCandless added a comment -

          Patch looks good:

          • Can you make sure CHANGES describes this new behavior (Field is
            allowed to have both a tokenStream and a String/Reader/binary
            value)?
          • The javadoc for readerValue is wrong (copy/paste from stringValue)
          • Can you spell out more clearly in the javadocs that even when a
            tokenStream value is set, one of String/Reader/binary may also be
            set, or, not, and if so, that "other" value is only used for
            stored fields. Eg, explain why one would use setTokenStream
            instead of setValue(TokenStream value).
          Show
          Michael McCandless added a comment - Patch looks good: Can you make sure CHANGES describes this new behavior (Field is allowed to have both a tokenStream and a String/Reader/binary value)? The javadoc for readerValue is wrong (copy/paste from stringValue) Can you spell out more clearly in the javadocs that even when a tokenStream value is set, one of String/Reader/binary may also be set, or, not, and if so, that "other" value is only used for stored fields. Eg, explain why one would use setTokenStream instead of setValue(TokenStream value).
          Hide
          Yonik Seeley added a comment -

          Patch attached.

          I've attempted to clean up some of the semantics of Fieldable - it's really a reasonable interface, with just a few redundancies (isBinary/getBinary). I've purposely avoided messing with Field more than necessary (adding yet more constructors, constants, etc).

          This change is back compatible as it just removes some unnecessary restrictions. I'll commit in a few days.

          Show
          Yonik Seeley added a comment - Patch attached. I've attempted to clean up some of the semantics of Fieldable - it's really a reasonable interface, with just a few redundancies (isBinary/getBinary). I've purposely avoided messing with Field more than necessary (adding yet more constructors, constants, etc). This change is back compatible as it just removes some unnecessary restrictions. I'll commit in a few days.
          Hide
          Yonik Seeley added a comment -

          committed.

          Show
          Yonik Seeley added a comment - committed.

            People

            • Assignee:
              Yonik Seeley
              Reporter:
              Yonik Seeley
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development