[PATCH] RussianAnalyzer's tokenizer skips numbers from input text,

• Key: LUCENE-1003
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Otis Gospodnetic
• Reporter: OpenTeam.ru
• Component/s: Analysis
• Affects Version/s: 2.2

Description

RussianAnalyzer's tokenizer skips numbers from input text, so that resulting token stream miss numbers. Problem can be solved by adding numbers to RussianCharsets.UnicodeRussian. See test case below for details.

TestRussianAnalyzer.java

public class TestRussianAnalyzer extends TestCase {

  Reader reader = new StringReader("text 1000");

  // test FAILS
  public void testStemmer() {
    testAnalyzer(new RussianAnalyzer());
  }

  // test PASSES
  public void testFixedRussianAnalyzer() {
    testAnalyzer(new RussianAnalyzer(getRussianCharSet()));
  }

  private void testAnalyzer(RussianAnalyzer analyzer) {
    try {
      TokenStream stream = analyzer.tokenStream("text", reader);
      assertEquals("text", stream.next().termText());
      assertNotNull(stream.next());
    } catch (IOException e) {
      fail(e.getMessage());
    }
  }

  private char[] getRussianCharSet() {
    int length = RussianCharsets.UnicodeRussian.length;
    final char[] russianChars = new char[length + 10];

    System
        .arraycopy(RussianCharsets.UnicodeRussian, 0, russianChars, 0, length);
    russianChars[length++] = '0';
    russianChars[length++] = '1';
    russianChars[length++] = '2';
    russianChars[length++] = '3';
    russianChars[length++] = '4';
    russianChars[length++] = '5';
    russianChars[length++] = '6';
    russianChars[length++] = '7';
    russianChars[length++] = '8';
    russianChars[length] = '9';
    return russianChars;
  }
}

Comments

[Nick Menere]

Yeah,
I raised this on the dev list a few months ago and didn't get much response.

I think I might even be responsible for that code above. It was meant more as hack to get a customer up and running.

Cheers,
Nick

[OpenTeam.ru]

Yeah, Nick, the code above was taken from your JIRA issue. We wasn't able to find similar issue in Lucene issue tracker. We're using Lucene a lot so we needed this bug fixed in the core.

[OpenTeam.ru]

Patch that adds numbers to RussianCharset
usage: patch RussianCharsets.java < RussianCharsets.java.patch

[Grant Ingersoll]

minor nit, can you add the test case to the patch as well?

TUSUR OpenTeam: would it be possible to get a unit test, too?

[Dmitry Lihachev]

Patch that adds new test to the TestRussianAnalyzer
usage:
patch TestRussianAnalyzer.java < TestRussianAnalyzer.java.patch