[LUCENE-1758] improve arabic analyzer: light8 -> light10

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.9
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 23/Jul/09 17:59
• Updated: 28/Aug/22 12:04
• Resolved: 05/Aug/09 18:27

Description

Someone mentioned on the java user list that the arabic analysis was not as good as they would like.

This patch adds the لل- prefix (light10 algorithm versus light8 algorithm).
In the light10 paper, this improves precision from .390 to .413
They mention this is not statistically significant, but it makes linguistic sense and at least has been shown not to hurt.

In the future, I hope openrelevance will allow us to try some more approaches.