Lucene - Core
  1. Lucene - Core
  2. LUCENE-4063

FrenchLightStemmer performs abusive compression of (arbitrary) repeated characters in long tokens

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.4, 4.0-ALPHA
    • Fix Version/s: 4.0-ALPHA
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      FrenchLightStemmer performs aggressive deletions on repeated character sequences, even on numbers.
      This might be unexpected during full text search.

      1. LUCENE-4063.patch
        3 kB
        Steve Rowe
      2. SOLR-3463.patch
        2 kB
        Tanguy Moal
      3. SOLR-3463.patch
        2 kB
        Tanguy Moal
      4. SOLR-3463.patch
        2 kB
        Tanguy Moal

        Activity

        Hide
        Tanguy Moal added a comment - - edited

        This patch implements the solution suggested by Robert Muir on the ML.

        Patch for lucene/solr trunk, generated from root directory.

        Show
        Tanguy Moal added a comment - - edited This patch implements the solution suggested by Robert Muir on the ML. Patch for lucene/solr trunk, generated from root directory.
        Hide
        Tanguy Moal added a comment -

        Updated patch to cover corner case (code also performs additional deletion of last character if it equals last character minus 1.

        Also added very minimal unit test (which exhibited the uncovered corner case)

        Show
        Tanguy Moal added a comment - Updated patch to cover corner case (code also performs additional deletion of last character if it equals last character minus 1. Also added very minimal unit test (which exhibited the uncovered corner case)
        Hide
        Steve Rowe added a comment -

        +1

        Tanguy, can you add a couple more tests? You should demonstrate that the deletion of repeated characters still works (with letter chars). Also, since there are two repetition removal operations in the code, a test specific to each would be useful.

        Show
        Steve Rowe added a comment - +1 Tanguy, can you add a couple more tests? You should demonstrate that the deletion of repeated characters still works (with letter chars). Also, since there are two repetition removal operations in the code, a test specific to each would be useful.
        Hide
        Tanguy Moal added a comment -

        Added some tests related to this issue.

        Show
        Tanguy Moal added a comment - Added some tests related to this issue.
        Hide
        Steve Rowe added a comment -

        Added some tests related to this issue.

        The new patch looks identical to the old patch - I don't see any new tests?

        Show
        Steve Rowe added a comment - Added some tests related to this issue. The new patch looks identical to the old patch - I don't see any new tests?
        Hide
        Tanguy Moal added a comment -

        I didn't drink anything yet but maybe it's time to begin

        Show
        Tanguy Moal added a comment - I didn't drink anything yet but maybe it's time to begin
        Hide
        Steve Rowe added a comment -

        Tanguy, since this is entirely a Lucene change, I've moved the issue's project from Solr to Lucene.

        Show
        Steve Rowe added a comment - Tanguy, since this is entirely a Lucene change, I've moved the issue's project from Solr to Lucene.
        Hide
        Steve Rowe added a comment -

        Patch with a couple more tests, and a CHANGES.txt entry.

        Committing to trunk shortly.

        Show
        Steve Rowe added a comment - Patch with a couple more tests, and a CHANGES.txt entry. Committing to trunk shortly.
        Hide
        Steve Rowe added a comment -

        Committed to trunk. Thanks Tanguy!

        I'm not sure if this should be committed on the 3.6 branch, since that branch is bug-fix only, and this issue is marked as an improvement. Thoughts?

        Show
        Steve Rowe added a comment - Committed to trunk. Thanks Tanguy! I'm not sure if this should be committed on the 3.6 branch, since that branch is bug-fix only, and this issue is marked as an improvement. Thoughts?
        Hide
        Tanguy Moal added a comment -

        I'd be glad to see this on 3.x x >=4 since that's the version I used to spot the issue, may be should I have marked this issue as a bug rather than improvement ?

        I have a custom filterfactory marking numbers as keywords anyway as I needed a quick fix.
        So from my point of view it doesn't really matter... I could just drop that filter from my analysis if the patch finds its way to 3x.

        Thank you very much for your quick responses about this issue.

        Show
        Tanguy Moal added a comment - I'd be glad to see this on 3.x x >=4 since that's the version I used to spot the issue, may be should I have marked this issue as a bug rather than improvement ? I have a custom filterfactory marking numbers as keywords anyway as I needed a quick fix. So from my point of view it doesn't really matter... I could just drop that filter from my analysis if the patch finds its way to 3x. Thank you very much for your quick responses about this issue.
        Hide
        Robert Muir added a comment -

        As far as this being a bug, the original code implements the algorithm it claims to implement, and undoubling anything was its heuristic: see http://members.unine.ch/jacques.savoy/clef/frenchStemmerPlus.txt

        Show
        Robert Muir added a comment - As far as this being a bug, the original code implements the algorithm it claims to implement, and undoubling anything was its heuristic: see http://members.unine.ch/jacques.savoy/clef/frenchStemmerPlus.txt
        Hide
        Steve Rowe added a comment -

        I agree with Robert, this is a design change, not a bug fix, so I won't backport to the 3.6 branch.

        Show
        Steve Rowe added a comment - I agree with Robert, this is a design change, not a bug fix, so I won't backport to the 3.6 branch.
        Hide
        Tanguy Moal added a comment - - edited

        I agree with both of you, it sounds like a design change.

        I think Jacques Savoy's algorithm was intended to be used on words. Not on numbers, or mixes of both (like in 22h00).

        Which is true for any stemmer, I think. That's why on the mailing I also suggested we could have each stemmer share a common interface that would filter non-stemmable literals out of the way. That could prevent the same issue to rise from a different stemming implementation.

        I'm just saying this as I think about it.

        Show
        Tanguy Moal added a comment - - edited I agree with both of you, it sounds like a design change. I think Jacques Savoy's algorithm was intended to be used on words. Not on numbers, or mixes of both (like in 22h00). Which is true for any stemmer, I think. That's why on the mailing I also suggested we could have each stemmer share a common interface that would filter non-stemmable literals out of the way. That could prevent the same issue to rise from a different stemming implementation. I'm just saying this as I think about it.
        Hide
        Robert Muir added a comment -

        That's why on the mailing I also suggested we could have each stemmer share a common interface that would filter non-stemmable literals out of the way

        We actually have this in place, but its too limited. Its called KeywordAttribute. When this is set, the stemmer will not touch the word.

        Currently the only way to set this out of box is to use KeywordMarkerFilter which takes a Set of protected words.

        But to make your idea more flexible: I could imagine a couple more filters:

        • one that marks as Keyword based on a set of types. In this case you would just add NUM to that set, and no stemmers would touch any numbers. Of course
          for french this is solved already, but imagine if you are using the URLEmail tokenizer: I think a set like { URL, EMAIL }

          would be very useful,
          otherwise stemmers will probably muck with them.

        • one that marks as Keyword based on a regular expression. This could be good for fine-tuning stemmers for a lot of general purpose needs: e.g. on the
          mailing list before someone was unhappy about how russian stemmers would treat russian place names and they had a certain set of suffixes they didnt
          want stemmed.

        Anyway, I would really like to see these filters, I think they would be pretty simple to implement as well.

        Show
        Robert Muir added a comment - That's why on the mailing I also suggested we could have each stemmer share a common interface that would filter non-stemmable literals out of the way We actually have this in place, but its too limited. Its called KeywordAttribute. When this is set, the stemmer will not touch the word. Currently the only way to set this out of box is to use KeywordMarkerFilter which takes a Set of protected words. But to make your idea more flexible: I could imagine a couple more filters: one that marks as Keyword based on a set of types. In this case you would just add NUM to that set, and no stemmers would touch any numbers. Of course for french this is solved already, but imagine if you are using the URLEmail tokenizer: I think a set like { URL, EMAIL } would be very useful, otherwise stemmers will probably muck with them. one that marks as Keyword based on a regular expression. This could be good for fine-tuning stemmers for a lot of general purpose needs: e.g. on the mailing list before someone was unhappy about how russian stemmers would treat russian place names and they had a certain set of suffixes they didnt want stemmed. Anyway, I would really like to see these filters, I think they would be pretty simple to implement as well.

          People

          • Assignee:
            Steve Rowe
            Reporter:
            Tanguy Moal
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development