[NUTCH-2276] Tika Boilerpipe Parser in combo with RSS items doesn't work

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.11, 1.12
• Component/s: parser

Environment

feed parser for RSS
Tika parser with boilerpipe (with ArticleExtractor) for HTML

• Assignee: Unassigned
• Reporter: Francesco Capponi
• Created: 08/Jun/16 22:06
• Updated: 08/Jun/16 22:06

Description

Sometimes it happens that the text (description) for an RSS item is too short or has characteristics that Tika with Boilerpipe decide to cut the entire text, resulting in an empty string.

in fact when the feed plugin selects a parser uses the function:
Parser parser = parserFactory.getParsers(contentType, link)[0];
the content being a HTML returns the Tika Boilerpipe article extractor.

Since the description text of an RSS as far as I know is always html, instead of asking for the contentType, we could set another mimetype for this specific case
String contentType = contentMeta.get(Response.CONTENT_TYPE);
->String contentType = "text/html-short";