Complete rewrite of
Total rewrite to a new modular implementation:
Removes old evaluator recommender implementation.
New RecommenderEvaluator interface with PreferenceBased and OrderBased implementations.
New SamplingDataModel wrapper supplies randomly selected prefs from a delegate DataModel.
PreferenceBaseRecommenderEvaluator does roughly what the old Abstract.....Evaluator does, but uses SamplingDataModel to implement hold-outs.
RecommenderEvaluator allows different calculation formula for evaluators. The different calculations from the first patch are picked with a choosable Enum.
I'm happy with that it does, and was able to analyze my recommender projects more effectively.
I'm not sure exactly what the old RecommenderEvaluator did with held-out sampled data. This code from GroupLensRecommenderEvaluatorRunner does the same thing, I think. The training datamodel holds out the given percentage of both users and preferences within the remaining users.
RecommenderEvaluator evaluator = new PreferenceBasedRecommenderEvaluator();
File ratingsFile = TasteOptionParser.getRatings(args);
DataModel model = ratingsFile == null ? new GroupLensDataModel() : new GroupLensDataModel(ratingsFile);
GroupLensRecommenderBuilder recommenderBuilder = new GroupLensRecommenderBuilder();
DataModel trainingModel = new SamplingDataModel(model, 0.0, 0.9, Mode.USER);
DataModel testModel = glModel;
Recommender trainingRecommender = recommenderBuilder.buildRecommender(trainingModel);
Recommender testRecommender = recommenderBuilder.buildRecommender(testModel);
RunningAverage tracker = new CompactRunningAverageAndStdDev();
evaluator.evaluate(trainingRecommender, testRecommender, 50, tracker, RecommenderEvaluator.Formula.NONE);
double average = tracker.getAverage();