Lucene - Core
  1. Lucene - Core
  2. LUCENE-4138

Update morfologik (polish stemming) to 1.5.3

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-BETA, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Just released. Updates to the dictionary but most of all – it comes with a clean BSD license (including dictionary data).

      1. LUCENE-4138.patch
        35 kB
        Dawid Weiss
      2. LUCENE-4138.patch
        33 kB
        Dawid Weiss
      3. LUCENE-4138.patch
        13 kB
        Dawid Weiss

        Activity

        Hide
        Dawid Weiss added a comment -

        This will be trickier than I initially thought because of data format changes in Morfologik. Tags have been normalized (expanded) so what previously returned:

        > WordData[liście,liść,subst:pl:acc.nom.voc:m3]
        > WordData[liście,list,subst:sg:loc.voc:m3]
        > WordData[liście,lista,subst:sg:dat.loc:f]

        now returns:

        > WordData[liście,liście,subst:sg:acc:n2]
        > WordData[liście,liście,subst:sg:nom:n2]
        > WordData[liście,liście,subst:sg:voc:n2]
        > WordData[liście,liść,subst:pl:acc:m3]
        > WordData[liście,liść,subst:pl:nom:m3]
        > WordData[liście,liść,subst:pl:voc:m3]
        > WordData[liście,list,subst:sg:loc:m3]
        > WordData[liście,list,subst:sg:voc:m3]
        > WordData[liście,lista,subst:sg:dat:f]
        > WordData[liście,lista,subst:sg:loc:f]

        which generates lots of identical base forms with a variable candidate tag. I think this will require patching to simulate previous behavior (single base form, multiple-tags attribute). I think modifying MorphosyntacticTagAttribute to carry a list of tags for the current form is a way to go. This would be a non-compatible change (which may just as well be good for 4.x).

        Show
        Dawid Weiss added a comment - This will be trickier than I initially thought because of data format changes in Morfologik. Tags have been normalized (expanded) so what previously returned: > WordData [liście,liść,subst:pl:acc.nom.voc:m3] > WordData [liście,list,subst:sg:loc.voc:m3] > WordData [liście,lista,subst:sg:dat.loc:f] now returns: > WordData [liście,liście,subst:sg:acc:n2] > WordData [liście,liście,subst:sg:nom:n2] > WordData [liście,liście,subst:sg:voc:n2] > WordData [liście,liść,subst:pl:acc:m3] > WordData [liście,liść,subst:pl:nom:m3] > WordData [liście,liść,subst:pl:voc:m3] > WordData [liście,list,subst:sg:loc:m3] > WordData [liście,list,subst:sg:voc:m3] > WordData [liście,lista,subst:sg:dat:f] > WordData [liście,lista,subst:sg:loc:f] which generates lots of identical base forms with a variable candidate tag. I think this will require patching to simulate previous behavior (single base form, multiple-tags attribute). I think modifying MorphosyntacticTagAttribute to carry a list of tags for the current form is a way to go. This would be a non-compatible change (which may just as well be good for 4.x).
        Hide
        Dawid Weiss added a comment -

        Work in progress update, tests not passing, filter not updated to accept new data.

        Show
        Dawid Weiss added a comment - Work in progress update, tests not passing, filter not updated to accept new data.
        Hide
        Dawid Weiss added a comment -

        Updated patch. Not backwards compatible (intentially): MorphosyntacticTagAttribute has been renamed to MorphosyntacticTagsAttribute (note plural) and now carries a list of tags for the current stem.

        Show
        Dawid Weiss added a comment - Updated patch. Not backwards compatible (intentially): MorphosyntacticTagAttribute has been renamed to MorphosyntacticTagsAttribute (note plural) and now carries a list of tags for the current stem.
        Hide
        Dawid Weiss added a comment -

        Updated patch with minor fixes (corrected module fileset, optimized buffer reuse for tags).

        Show
        Dawid Weiss added a comment - Updated patch with minor fixes (corrected module fileset, optimized buffer reuse for tags).
        Hide
        Dawid Weiss added a comment -

        If there are no objections I'll commit this shortly.

        Show
        Dawid Weiss added a comment - If there are no objections I'll commit this shortly.
        Hide
        Hoss Man added a comment -

        hoss20120711-manual-post-40alpha-change

        Show
        Hoss Man added a comment - hoss20120711-manual-post-40alpha-change

          People

          • Assignee:
            Dawid Weiss
            Reporter:
            Dawid Weiss
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development