[MAHOUT-955] Bayes classification result are unstable after classifying non-existing features

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.5
• Fix Version/s: 0.7
• Labels: Bayes, InMemoryDatastore, vocabCount

Environment

JRE 7

• Assignee: Robin Anil
• Reporter: Michał B
• Created: 24/Jan/12 18:47
• Updated: 31/Jan/24 22:12
• Resolved: 04/Jun/12 14:08

Description

Bayes classification results are unstable, and change during runtime!

Sample test:
MyClassifier classifier = new MyClassifier(new BayesAlgorithm(),params); //Custom simple wrapper for classifier
ClassifierResult category = classifier.classify("existing");
double resultA = category.getScore();
category = classifier.classify("nonexisting");
category = classifier.classify("existing");
double resultB = category.getScore();
Assert.assertEquals(resultA,resultB,0.0); // FAIL!!!

Test like the one above will fail. Because nonexisting tokens are added to InMemoryBayesDatastore->featureDictionary therefore datastore.getWeight("sumWeight", "vocabCount") change after classification of unknown feature. Moreover, the featureDictionary fills with not wanted strings using heapspace.

More on this here
http://www.lucidimagination.com/search/document/7dabe3efec8d136d/issues_with_memory_use_and_inconsistent_or_state_influenced_results_when_using_cbayesalgorit#8853165db260bf75