[STANBOL-197] Enhancement Engine for Wikipedia/DBpedia-based topic classification of text content

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Enhancement Engines, Entityhub
• Labels: text-categorization
• Assignee: Rafa Haro
• Reporter: Olivier Grisel
• Created: 12/May/11 13:47
• Updated: 25/Jun/15 17:13
• Resolved: 25/Jun/15 17:13

Description

Implementation plan:

Use MoreLikeThis queries on a SolrYard instance with topics indexed by aggregating the text of abstracts of all entities marked categorized by a given SKOS topic from DBpedia.

Such an index can be constructed using the pig scripts available at:
https://github.com/ogrisel/pignlproc/tree/master/examples/topic-corpus
or
https://github.com/ogrisel/dbpediakit

In order to perform MoreLikeThis queries using the SolrJ API it is possible to do the following:

#1 - Define the mlt handles in solrconfig.xml (it's not defined in the example
solrconfig.xml I was using):

<requestHandler name="/mlt" class="solr.MoreLikeThisHandler" />

#2 - with Solrj, access the mlt handler via something similar to the following:

query.setQueryType("/" + MoreLikeThisParams.MLT);
query.set(MoreLikeThisParams.MATCH_INCLUDE, false);
query.set(MoreLikeThisParams.MIN_DOC_FREQ, 1);
query.set(MoreLikeThisParams.MIN_TERM_FREQ, 1);
query.set(MoreLikeThisParams.SIMILARITY_FIELDS, "subject,body");
query.setQuery("Your query here or in my case the unique key field:value");

Issue Links

• is superceded by: STANBOL-1294 Topic Classification Framework for Stanbol
• relates to: STANBOL-92 package solr index for popular entities
• relates to: STANBOL-617 Define how TopicEnhancements are written to the Enhancement Structure
• relates to: STANBOL-28 Extend the RDF enhancement vocabulary to handle topics explicitly

Sub-Tasks

• Pig script to extract the list of all sentences where an entity / topic occur in the body of a wikipedia article: Closed
• Integrate pignlproc outpout (TSV or other format) with the Stanbol indexing tools: Closed
• Extend SolrYard to deal with similarity queries: Closed
• Implement incremental model update for the TopicClassifierEngine: Closed
• Use negative examples and DF vector to prune topic model and increase the precision: Closed
• Implement cross validated evaluation of the classifier with Precision / Recall and F1 score for each topic in the model: Closed
• Add web interface to train a classifier and document the REST API: Open