Solr
  1. Solr
  2. SOLR-2982

Upgrade Apache Commons Codec to version 1.6 in order to add new Beider-Morse Phonetic Matching (BMPM) option

    Details

      Description

      Apache Commons Codec released version 1.6 of their codec pack in November, 2011. Along with a few bug fixes, 1.6 contains a great new phonetic matching system called Beider-Morse Phonetic Matching (BMPM) that is far superior to the existing phonetic codecs, such as regular soundex, metaphone, caverphone, and so on. BMPM has actually been available for some time, but this is the first port of it to java, and its first commit in the Apache ecosystem.

      For a lot more information, see here: http://stevemorse.org/phoneticinfo.htm and http://stevemorse.org/phonetics/bmpm.htm

      BMPM would be a fantastic "soundalike" tool to help search for personal names (or just surnames) in a Solr/Lucene index, much better than Levenshtein distance for this use case.

      1. SOLR-2982.patch
        19 kB
        Robert Muir

        Issue Links

          Activity

          Hide
          Robert Muir added a comment -

          Attached is a patch, really bmpm needs its own filter because this encoding is shoved onto the commons-codec API (but imo this is really confusing: it doesnt really make sense to use strings here)

          the output of this thing is actually syntax such as (((x|y|z)-(a|b..., which means we have to parse it again to do anything with it.

          I also noticed this new encoder seems to have performance issues, i had to scale back the random strings test somewhat.

          Show
          Robert Muir added a comment - Attached is a patch, really bmpm needs its own filter because this encoding is shoved onto the commons-codec API (but imo this is really confusing: it doesnt really make sense to use strings here) the output of this thing is actually syntax such as (((x|y|z)-(a|b..., which means we have to parse it again to do anything with it. I also noticed this new encoder seems to have performance issues, i had to scale back the random strings test somewhat.
          Hide
          Brooke Schreier Ganz added a comment -

          Thank you so much for working on this! I just tested out the latest nightly build on my laptop and everything works great.

          I use Solr to run a non-profit group's 190,000+ record genealogy database and this new ability to do proper soundalike surname searches through our listings of consonant-heavy Central and Eastern European surnames (and their multitudes of "creative" spelling variants) will make things a lot easier. You just made a lot of genealogists very happy.

          Thanks again!

          Show
          Brooke Schreier Ganz added a comment - Thank you so much for working on this! I just tested out the latest nightly build on my laptop and everything works great. I use Solr to run a non-profit group's 190,000+ record genealogy database and this new ability to do proper soundalike surname searches through our listings of consonant-heavy Central and Eastern European surnames (and their multitudes of "creative" spelling variants) will make things a lot easier. You just made a lot of genealogists very happy. Thanks again!
          Hide
          Thomas Champagne added a comment -

          I noticed too the bad performance of Beider Morse encoder. So, I have created an issue CODEC-174 in the commons-codec project to improve the performance. Currently, I have created two patches that allow dividing the encoding time by 2.
          If you want a better Beider Morse encoder, you can join us on the issue CODEC-174

          Show
          Thomas Champagne added a comment - I noticed too the bad performance of Beider Morse encoder. So, I have created an issue CODEC-174 in the commons-codec project to improve the performance. Currently, I have created two patches that allow dividing the encoding time by 2. If you want a better Beider Morse encoder, you can join us on the issue CODEC-174

            People

            • Assignee:
              Unassigned
              Reporter:
              Brooke Schreier Ganz
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development