Solr
  1. Solr
  2. SOLR-876

Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4
    • Component/s: search
    • Labels:
      None

      Description

      Add ability to optionally splitOnNumerics WordDelimiterFilter/Factory

      Default behaviour is to splitOnNumerics as WordDelimiterFilter/Factory does now

      I was having issues with e.g. Java/J2SE becoming split into tokens 'Java','J' '2' and 'SE' which isn't desired behavior in my instance, I wanted this to be tokens 'Java', 'J2SE'. Another option I thought about but not implemented was to have a protected list of words like solr.EnglishPorterFilterFactory

      1. SOLR-876.patch
        15 kB
        Dan Rosher
      2. SOLR-876.patch
        19 kB
        Dan Rosher
      3. SOLR-876.patch
        10 kB
        Dan Rosher

        Activity

        Hide
        Dan Rosher added a comment -

        Add ability to splitOnNumerics [default = 1 as is current behaviour.]

        Also some additional tests in src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java and changes to src/test/test-files/solr/conf/schema.xml

        Show
        Dan Rosher added a comment - Add ability to splitOnNumerics [default = 1 as is current behaviour.] Also some additional tests in src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java and changes to src/test/test-files/solr/conf/schema.xml
        Hide
        Dan Rosher added a comment -

        Added in ability to 'protect' words against further tokenizing

        Show
        Dan Rosher added a comment - Added in ability to 'protect' words against further tokenizing
        Hide
        Hoss Man added a comment -

        Dan: reading over your patch it seems that several test* methods were removed from TestWordDelimiterFilter.java, as well as change some parts of the test to use the deprecated Token.termText() method (even though the test already uses Token.termBuffer())

        Also: for back compatibility we should leave the existing WDF constructors alone, and either add a new one with the new args, or make the new options configured via setters.

        Show
        Hoss Man added a comment - Dan: reading over your patch it seems that several test* methods were removed from TestWordDelimiterFilter.java, as well as change some parts of the test to use the deprecated Token.termText() method (even though the test already uses Token.termBuffer()) Also: for back compatibility we should leave the existing WDF constructors alone, and either add a new one with the new args, or make the new options configured via setters.
        Hide
        Dan Rosher added a comment -

        Thanks for reading over it Hoss, I think I originally made the patch against a previous version of the solr trunk, the test omissions weren't intentional, I should have done a better job of reading the patch!

        I've made new constructors, and deprecated the current/older ones, not sure if this is the approach you recommend though.

        Show
        Dan Rosher added a comment - Thanks for reading over it Hoss, I think I originally made the patch against a previous version of the solr trunk, the test omissions weren't intentional, I should have done a better job of reading the patch! I've made new constructors, and deprecated the current/older ones, not sure if this is the approach you recommend though.
        Hide
        Hoss Man added a comment -

        Thanks for the great patch Dan.

        Committed revision 727677.

        (i also added a few more tests to prove to myself that protwords was really working)

        Show
        Hoss Man added a comment - Thanks for the great patch Dan. Committed revision 727677. (i also added a few more tests to prove to myself that protwords was really working)
        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4

          People

          • Assignee:
            Hoss Man
            Reporter:
            Dan Rosher
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development