[TEXT-155] Add a generic OverlapSimilarity measure

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Implemented
• Affects Version/s: 1.6
• Fix Version/s: 1.7
• Assignee: Alex Herbert
• Reporter: Alex Herbert
• Created: 07/Mar/19 13:42
• Updated: 13/Mar/19 16:25
• Resolved: 13/Mar/19 16:25

Description

The SimilarityScore<T> interface can be used to compute a generic result. I propose to add a class that can compute the intersection between two sets formed from the characters. The sets must be formed from the CharSequence input to the apply method using a Function<CharSequence, Set<T>> to convert the CharSequence. This function can be passed to the SimilarityScore<T> during construction.

The result can then be computed to have the size of each set and the intersection.

I have created an implementation that can compute the equivalent of the JaccardSimilary class by creating Set<Character> and also the F1-score using bigrams (pairs of characters) by creating Set<String>. This relates to Text-126 which suggested an algorithm for the Sorensen-Dice similarity, also known as the F1-score.

Here is an example:

// Match the functionality of the JaccardSimilarity class
Function<CharSequence, Set<Character>> converter = (cs) -> {
    final Set<Character> set = new HashSet<>();
    for (int i = 0; i < cs.length(); i++) {
        set.add(cs.charAt(i));
    }
    return set;
};
IntersectionSimilarity<Character> similarity = new IntersectionSimilarity<>(converter);
IntersectionResult result = similarity.apply("something", "something else");

The result has the size of set A, set B and the intersection between them.

This class was inspired by my look through the various similarity implementations. All of them except the CosineSimilarity perform single character matching between the input CharSequence}}s. The {{CosineSimilarity tokenises using whitespace to create words.

This more generic type of implementation will allow a user to determine how to divide the CharSequence but to create the sets that are compared, e.g. single characters, words, bigrams, etc.

Issue Links

• is related to: TEXT-126 Dice's Coefficient Algorithm in String similarity
• links to: GitHub Pull Request #109